Gentoo Archives: gentoo-portage-dev

From: Brian Harring <ferringb@g.o>
To: gentoo-dev@l.g.o
Cc: gentoo-portage-dev@l.g.o
Subject: [gentoo-portage-dev] Re: [gentoo-dev] EBUILD_FORMAT support
Date: Fri, 26 Aug 2005 07:38:45
Message-Id: 20050826073529.GP1701@nightcrawler
1 Pardon the delay, been putting this one off since it's going to be a
2 fun one to address, and will be a bit long :)
4 On Thu, Aug 25, 2005 at 12:34:00PM +0200, Paul de Vrieze wrote:
5 > What I mean is compatibility with current portage versions. Current
6 > versions do not understand EAPI. There would be a good chance that they
7 > could choke on packages with all kinds of new features, even in the sync
8 > phase. A different extension would ensure that those portage versions
9 > would still work (crippled) on a new tree. Of course such an extension
10 > change should only be done once. Once the API versions are available this
11 > is not an issue.
13 General portage stance towards EAPI is unset EAPI == 0 (current stable
14 ebuild format); if EAPI > then portage internal EAPI, unable to merge,
15 which should be able to be detected during buildplan.
17 Current portage doesn't know about EAPI; boned in that respect I'll
18 admit, but it's the case for all new features rolled out- three options
19 for dealing with this
20 1) Usual method, deploy support, N months later use support.
21 2) tweak stable so it's aware and can complain. Still requires
22 people to upgrade, just makes it so that they're not forced into
23 upgrading to 3.x; this is mainly a benefit for those who may don't
24 care to try the first few releases of 3.x when it hits (akin to
25 people dodging the first release or two of a gcc release).
27 Worth noting that one rather visibile aspect of EAPI=1 is that
28 (assuming the council votes on it, and yay's it) glep33 *will* result
29 in current eclasses being effectively abandoned w/in the N months
30 after an EAPI capable portage is released.
32 Sound kind of bad, but people will have to upgrade for the
33 capabilities. If EAPI was pegged into portage/ebuilds already
34 it wouldn't be an issue, issues could be detected prior.
35 Unfortunately it's not, and introduction of it (and use of it) is
36 going to involve a few road bumps.
38 Plus side, once it's in, portage *will* know if the ebuild is
39 incompatible with the pythonic/bash ebuild code, and portage/the UI
40 can act accordingly.
42 Meanwhile, the changes that are being pushed into EAPI are addition of
43 configure phase (broken out from compile), elib addition, and eclass2
44 support (same beast, different rules due to env save/restoration).
46 The potential for horkage on sync'ing isn't there due to the fact
47 that's purely python side; ebuild*sh doesn't play into it.
49 Re: regen, issue isn't really there either; if you try and merge an
50 eapi=0 on a non eapi aware portage, it works, same as it did before.
51 If you try to merge an eapi=1 ebuild you hit either an issue with
52 inherit, or a bail immediately in src_compile, due to the fact eapi=1
53 ebuilds will seperate configure out from compile (eapi=0 portage won't
54 know to call it; no configure == failed compile).
56 That said, there also quite likely is a change coming down the pipe to
57 the tree's cache; the change will shift the rsync'd metadata cache
58 over to a key/val based cache.
60 Why oh why, yet another cache change? Simple. The change moves away
61 from list based format to key:value pairs; in short it's a change that
62 once made, means keys can be added to the cache from that point on
63 without causing cache complaints on sync'ing. Last cache breakage, I
64 swear :P
66 EAPI addition being the next key tagged in; stable (not surprising)
67 needs to be released with a version capable of reading both old and
68 new format; once that's done, time for the usual "yo, upgrade people,
69 something's coming down the line". Same version that supports
70 old and new cache format can also include rudimentary eapi awareness.
72 At least that's what I'm thinking. It's roughly inline with the
73 previous forced cache breakages, just in this case slipping in some
74 extra support in the process.
76 Notices obviously would go out prior to moving on this also, along
77 with a good chunk of waiting.
80 > > > ps. I would also suggest requiring that EAPI can be retrieved by a
81 > > > simple line by line parsing without using bash. (This allows for
82 > > > changing the parsing system)
83 > >
84 > > No, that yanks EAPI setting away from eclasses.
85 >
86 > If the eclasses follow similar rules that would be easilly parseable.
87 > (taking inherit ...) into account is easy as long as the inherit line is
88 > on one line of it's own. (unconditionally) These rules that would
89 > allready be followed out of style reasons would make various kinds of
90 > parsers able to parse them.
92 while it's insane, people *can* use indirection (eg inherit $var) for
93 inherit's as long as it's deterministic, always the same inherit call
94 for that ebuild's data. Don't see a good reason to ixnay that, which
95 means we'd have to parse the whole enchilada, eclasses and the ebuild.
97 Effectively, raiding a single var out wouldn't fly; eclasses could
98 override an ebuild's eapi setting for example, just like any other
99 metadata key (imo).
101 A *true* format change, moving away from bash for example or moving to
102 an executing design of ebuilds would require an extension change; such
103 a change must imo anyways, since it's not a change of the ebuild env's
104 template/hooks; either it's a fundamentally different model for
105 ebuilds- either via no longer being bash based, or moving away from our
106 declarative design of ebuilds.
109 > > Only time this would be required is if we move away from bash; if that
110 > > occurs, then I'd think a new extension would be required.
111 <inserting a comment> contradicting myself via above, above is correct </comment>
112 >
113 > It would allow to for example restrict the ebuild format such that initial
114 > parsing is not done by bash (but the files are still parseable by bash).
115 > If we perform changes I think it should be done right in the first place.
116 Elaborate please
118 > > As is, shifting the 'template' loaded for an ebuild can be done in
119 > > ebd's init_environ easy enough, so no reason to add the extra
120 > > restrictions/changes.
121 >
122 > One of the issues of ebuilds is the cache/metadata stuff. Parsing an
123 > ebuild for basic information takes a lot of time. This can be done lots
124 > faster with a less featured parser (I've written one some day) that
125 > accepts 98% of all current ebuilds, just doesn't like dynamic features in
126 > the toplevel. Such a parser could be a python plugin and as such easy to
127 > use from python. However to ensure compatibility with a faster parser the
128 > EAPI variable should be there in a way that is a little more strict than
129 > the other variables. And such a restriction is in practice not a
130 > restriction.
132 Any parser that doesn't support full bash syntax isn't acceptable from
133 where I sit; re: slow down, 2.1 is around 33% faster sourcing the
134 whole tree (some cases 60% faster, some 5%, etc). The speed up's are
135 also what allow template's to be swapped, the eapi concept.
137 I'd note limiting the bash capabilities is a restriction that
138 transcends anything EAPI should supply; changes to what's possible in
139 the language (a subset of bash syntax as you're suggesting) are a
140 seperate format from where I draw the line in the sand.
142 Mainly, limiting the syntax has the undesired affect of deviating from
143 what users/devs know already; mistakes *will* occur. QA tools can be
144 written, but people are fallable; both in writing a QA tool, and
145 abiding by the syntax subset allowed.
148 > The restriction I propose would be:
149 > - If EAPI is defined in the ebuild it should be unconditional, on it's own
150 > line in the toplevel of the ebuild before any functions are defined.
151 > (preferably the first element after the comments and whitespace)
152 >
153 > - If EAPI is not defined in the ebuild, but in an eclass, the inherit
154 > chain should be unconditional and direct. Further more in the eclass the
155 > above rules should be followed.
156 >
157 > Please note that many of the conditions are allready true for current
158 > ebuilds, just portage can "handle" more.
160 inherit chain must be unconditional anyways. re: eapi placement, I
161 would view that as somewhat arbitrary; the question is what gain it
162 would give.
164 I'd wonder about the parsing speed of your parser; the difference
165 between parsing ebuilds and running from cache metadata is several
166 orders of magnitude differant- the current cache backend flat_list
167 and portage design properly corrected ought to widen the gap too.
168 General cache lookup is slow due to-
169 A) bad call patterns, allowed by the api; N calls to get different
170 bits of metadata from a cpv, resulting in potentially N to disk set
171 of ops.
172 B) default cache requires opening/closing a file per cpv lookup; syscall's
173 are killer here.
174 C) every metadata lookup incurs 2 stats, ebuild and cache file.
176 Getting to the point; cache is 100x to 400x faster then sourcing for
177 <=2.0.51. Haven't tested it under 2.1, should be different due to
178 cache and regen fixups/rewrites.
180 Back to the point, essentially, EAPI matters in two places;
181 1) metadata transfer from the ebuild env into python side during
182 depends phase; has to know what to transfer key wise.
183 2) actual ebuild build phase executions; if it isn't the depends phase,
184 eapi being required so that the parser can swap drop in the appropriate
185 ebuild env template.
187 The restrictions suggested for EAPI would only make sense if eyeing
188 #1, an alternative parser; no reason to drop the cache unless the
189 parser is capable of hitting the same runtime performance the cache
190 can hit (frankly, it's not possible from where I'm sitting although
191 the gap can be narrowed).
193 So... the EAPI limitations, not much for due to the conclusion above.
195 Interested in the parser however, since ebd is effectively a pipe
196 hack so that pythonic portage can control I (and others)
197 have been after a bashlib for a while, just no one has crunched down
198 and done it (easier said then done I suspect).
200 My 2 cents at least.
201 ~harring


Subject Author
Re: [gentoo-portage-dev] Re: [gentoo-dev] EBUILD_FORMAT support Paul de Vrieze <pauldv@g.o>