Gentoo Archives: gentoo-dev

From: Brian Harring <ferringb@×××××.com>
To: Zac Medico <zmedico@g.o>
Cc: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] RFD: EAPI specification in ebuilds
Date: Mon, 12 Mar 2012 08:37:14
Message-Id: 20120312083612.GF7579@localhost
In Reply to: Re: [gentoo-dev] RFD: EAPI specification in ebuilds by Zac Medico
1 On Sun, Mar 11, 2012 at 09:08:24PM -0700, Zac Medico wrote:
2 > On 03/11/2012 06:55 PM, Brian Harring wrote:
3 > > On Sat, Mar 10, 2012 at 08:06:50AM -0800, Zac Medico wrote:
4 > >> Yeah. Another way of putting it is that the requirement to spawn a bash
5 > >> process and source the ebuild adds a ridiculous amount of unnecessary
6 > >> complexity, in violation of the KISS principle [1].
7 > >
8 > > This statement is incorrect.
9 > >
10 > > Even if EAPI could be parsed via some non sourcing approach, we
11 > > *still* have to source the ebuild to get the metadata for when the
12 > > EAPI is supported (the vast majority of usage). That complexity is
13 > > there one way or another- we wouldn't be trying to extract the EAPI
14 > > from the ebuild unless the cache was invalid/missing.
15 >
16 > There are a couple of other cases worth considering:
17 >
18 > 1) User downloads an overlay that doesn't provide cache. We want the
19 > package manager to give a pretty "EAPI unsupported" message, rather than
20 > spit out some bash syntax errors.
21
22 This criticsm pretty much applies *strictly to the existing
23 implementation*. It's disenguous busting it in this fashion.
24
25 EAPI as a function explicitly gives it an out before hitting any of
26 that, eliminating your entire critique. Same goes for parsing it out
27 of the ebuild, or renaming the extension.
28
29
30 > 2) You're assuming that a package manager can validate cache for an EAPI
31 > that it doesn't necessarily support.
32
33 Actually, I'm not.
34
35 > That's a somewhat fragile
36 > assumption, given the complexities of cache validation, which involve
37 > verification all files that affect metadata and those files may vary
38 > depending on the EAPI.
39
40 This is a fair bit of handwavey bullshit. The same "complexities of
41 cache validation" we have to by default deal with for valid/supported
42 EAPIs; the case for unsupported EAPIs is actually simpler than
43 for supported EAPis, and up until 7ddb7d30, was likely working fine in
44 portage. Tweaking portage to restore that support is pretty easy.
45
46 Also note that with the sole exception of g55, every implementation
47 has to deal with this specific issue, even g55 isn't fully exempt from
48 it since in cleansing a cache, there is the open question of pruning
49 cache entries for ebuilds known, but not understood by the local PM.
50
51 Either way, the algo is as follows: PM pulls EAPI from the ebuild
52 (regardless of method); PM recognizes that it can't handle it- thus
53 it stores EAPI:-$EAPI, and the normal cache ebuild checksumming;
54 whether it be mtime, md5 for md5-dict, or whatever form new caches
55 choose to use.
56
57 For attempts to load, the cache subsystem checks that validation
58 before trusting the cache entry- this is in place and has been for a
59 very, very long time. This isn't anything new.
60
61 So... the entry point is the ebuild; we have checksums for it, and a
62 prefixed versions of it's EAPI stored. The sole deal breaker here is
63 if we were to decide to allow eclasses to set EAPI (rather than the
64 current "you're not supposed to, although we don't explicitly block
65 it"- which could be tightened to be disallowed at the implementation
66 level for EAPI>4).
67
68 The following scenarios exist:
69
70 1) PM still doesn't support that EAPI, looks at the cache/ebuild:
71 checksums are the same, thus the stored EAPI is trustable, leading to
72 the PM knowing it still can't process that ebuild and masking it
73 appropriately.
74
75 2) PM still doesn't support that EAPI, looks, sees that the checksums
76 no longer match the ebuild. Re-pulls metadata/EAPI; if the ebuild has
77 been changed to a supported EAPI, continues on its way. If not,
78 stores the checksum/negated EAPI, and masks appropriately.
79
80 3) PM now supports that EAPI, looks, sees the negated eapi and
81 recognizes it as one it knowns, and forces regeneration.
82
83 All of those scenarios are there, and easy enough to handle- this
84 applies for the existing EAPI implementation additionally.
85
86
87 Now, this isn't to say that someone can't go and try to structure new
88 features explicitly to defeat the existing validations. They could.
89
90 The question here is whether or not new features would actually
91 *break* that w/out intentionally going out of their way to do so (aka,
92 skip the strawman arguments, actual examples required).
93
94
95 > > Phrasing it more bluntly: you can only avoid the sourcing step if you
96 > > can isolate that the EAPI is unsupported (which is extremely rare in
97 > > terms of actual user experience). For the rest of the time (well past
98 > > the 99% mark) sourcing is the immediate step following.
99 >
100 > For the sake of being robust in all possible cases, it's just a lot
101 > simpler if we can obtain the EAPI simply and reliably, without spawning
102 > bash to source the ebuild.
103
104 This still is arguable; any performant PM is going to have to do the
105 cache dance I mentioned above, or maintain an in memory list of
106 ebuilds it knows it can't handle (which if you want to talk robust,
107 fails miserably if the API isn't designed for sync invalidation of
108 that global list). Complexity, oh my.
109
110 That's the kicker from where I'm sitting; you're so focused on
111 avoiding bash that you ignore the complexities surrounding it that are
112 unavoidable regardless.
113
114 The EAPI function route resolve this issue w/out penalizing the normal
115 (aka supported) pathways, and does so at the same robustness as
116 everything sans g55 (which aims to allow us to use things other
117 than bash such as the mocked xml ebuild format). Frankly, the
118 function route gives us a bit more robustness- rather than allowing
119 the scenario of a PM that doesn't parse seeing a version that requires
120 it, via the function route doing `eapi 5 || die` has no such failure
121 mode.
122
123
124 > > Also, stop referencing wikipedia. People know what "trivial
125 > > objection" and "KISS" is.
126 >
127 > You can't assume that. On this list we've got potential to have readers
128 > and responders with lots of different backgrounds.
129
130 You might want to include links to Linux, gentoo, and bash while
131 you're at it.
132
133 Ya know, just in case.
134
135
136 > Your insistence on
137 > using bash to obtain the EAPI would make me wonder if you understood the
138 > KISS principle, if I didn't know you better.
139
140 Zing.
141
142 What I'd like to see, is accuracy in this discussion. Skip the
143 handwavey "complexity! complexity! complexity!" crap, same for
144 selective robustness definitions. Past attempts at this discussion
145 mostly failed due to people pulling crap like this and frankly it just
146 pisses people off.
147
148 Like it or not, each solution has technical pros/cons, more
149 importantly, it has an impact on developers meaning what they *like*
150 carries weight (rather than what you want them to do).
151
152 Shebang/parsing EAPI w/out bash, EAPI as a function, all are reliant
153 on the same cache mechanism. That's fact. G55 is the sole exception.
154
155 The question of what mechanism to use is dependant on what people
156 actually want, and how far we're willing to stretch the ebuild format
157 (versus defining a new one rather than new versions of it).
158
159 For the lifespan of gentoo thus far, our format has been bash; trying
160 to shift away from bash is fine, but as long as it *is* bash we should
161 be designing w/ that flow, including what is least likely to have a
162 developer screw up. Pragmatic reality, especially since we're still
163 stuck w/ compatibility issues getting to the new mechanism.
164
165
166 Frankly I'm starting to think I should just write up a table of
167 pros/cons, extending what Ulm has started; in particular, including
168 accurate statement of the transition costs of each (something people
169 seem to be ignoring- compatibility mechanisms are *not* cheap to get
170 in place).
171
172 ~harring

Replies

Subject Author
Re: [gentoo-dev] RFD: EAPI specification in ebuilds Ciaran McCreesh <ciaran.mccreesh@××××××××××.com>
Re: [gentoo-dev] RFD: EAPI specification in ebuilds Zac Medico <zmedico@g.o>