Gentoo Archives: gentoo-dev

From: Brian Harring <ferringb@×××××.com>
To: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] RFC about another *DEPEND variable
Date: Sat, 30 Sep 2006 19:38:44
Message-Id: 20060930193430.GA9496@seldon
In Reply to: Re: [gentoo-dev] RFC about another *DEPEND variable by Mike Frysinger
1 On Sat, Sep 30, 2006 at 02:01:08PM -0400, Mike Frysinger wrote:
2 > On Wednesday 27 September 2006 03:54, Brian Harring wrote:
3 > > > > Bleh, this is getting back to exactly my point that it's unbounded
4 > > > > resolution. To support this, every step of execution would require
5 > > > > scanning for dangling nodes to punt; aside from invalidating -p's
6 > > > > output it *still* is a collision-protect hit.
7 > > >
8 > > > when the package upgrade detects an ABI change you recalculate the
9 > > > packages that need to be rebuilt ...
10 > >
11 > > Reiterating, this screws over any form of up front calculation;
12 > > dialup users/per minute connections can't rely on emerge -f to
13 > > actually fetch all involved sources, -p results ain't guranteed
14 > > valid, parallelization must now lock at each threads merge on the off
15 > > chance a recalc is forced.
16 >
17 > it does indeed prevent full up front calculation. we can always make the
18 > behavior configurable; revdep on the fly or allow people to break it up. the
19 > key being that their system will still continue to function as the ABI lib
20 > has been preserved automagically
21
22 Folks aren't that daft; you make it an optional component, it means a
23 *proper* solution will never be pushed because the duct tape is in
24 place already.
25
26 If that's what folks want, sure, but what you're proposing is just
27 sliding NEEDED in as the defacto solution without labeling it as such.
28
29
30 > > > dangling nodes get recorded in the new package and considering these old
31 > > > files are not detrimental to the health of the system, you can do such a
32 > > > cleanup once at the end of the emerge
33 > >
34 > > It's not orphaning files, but your scheme doesn't work for any form of
35 > > interruption; failed builds, user intervention, etc, they all can
36 > > leave the lib stuck in the new contents.
37 >
38 > huh ? i outlined in a previous e-mail how by simply ordering the operations
39 > sanely, there is no race condition
40
41 Re-read your emails, and mine please. The scenario I pointed out was
42 ctrl+c'ing the merge while it's doing a rebuild for lib1 to lib2.
43
44 Now how does portage know that it needs to complete that upgrade for
45 the next emerge action? How does it know to even scan for lib1, let
46 alone punting?
47
48 It's *not* simple, and I keep pointing out this issue (and you're
49 missing it every damn time). Please address it, or point to where you
50 have (gone over the thread and still not seeing anything even
51 remotely touching this flaw).
52
53
54 > > Dealing with that possibility means the manager has to maintain on
55 > > disk a list of the refcount of each dangling libs to decrement,
56 > > unmerge has to modify said list, etc.
57 >
58 > which is already being done in the NEEDED file ... funny how unpainless it is
59 > to generate that file
60
61 First of all, unpainless is painful. Which is apt, actually.
62
63 Familiar with old style virtuals? That whole, "you have to walk the
64 whole vdb to collect all old style virtuals" ?
65
66 This is the same damn thing.
67
68 There is no refcount maintained via NEEDED, just a list of libs a
69 binary uses; you _still_ have to walk the damn vdb to build the
70 refcount list.
71
72 Now either you're forcing a fairly huge mapping in memory, or you're
73 forcing a scan of the vdb for every pkg operation that has NEEDED
74 entries it will install.
75
76 So no, NEEDED doesn't cover this, and my point still stands.
77
78
79 > > > > It also involves changing vdb nodes from "installed and usable" to
80 > > > > "installed/usable" or "lingering"
81 > > >
82 > > > no ... the old versions get merged into the new one as their existence is
83 > > > purely hidden
84 > >
85 > > How exactly are they going to be hidden? A new file? If so,
86 > > backwards compatibility for vdb for transitioning in such a support
87 > > has to be addressed.
88 >
89 > purely hidden from the standpoint of any new package trying to use it ...
90 > since you're only installing the runtime ABI library, you cannot link against
91 > it or utilize it any way other than existing files.
92
93 Except for dlopen, but thats again besides my original point; how do
94 you note to portage that the lib is one to watch so it can be punted?
95
96 You're suggesting it get shoved into contents, and for portage to
97 identify it, it has to do a revdep mapping on the fly to find it.
98
99 This *sucks* massively, both from a speed and complexity standpoint;
100 further, the lib isn't hidden from pkg ops (say quickpkging, or
101 binpkging) in any way so the cruft spreads. That's surprisingly minor
102 in comparison to implications of relying on refcounting to identify
103 the lib to punt.
104
105 If the lib to punt isn't tracked in some fashion, the only algo
106 you're left with is attempting to find all libs that have a refcount
107 of zero, and punt those- obviously, this is going to screw up just
108 about every single dlopen based linkage, literally, suck an algo
109 won't spot that python dlopen's it's modules, and would think the
110 refcount was zero (thus the so could get booted).
111
112 The response there is to add blacklists, directories to *not* inspect,
113 which is a further hack to try and make this tank fly.
114
115 *OR*, you're stuck maintaining a seperated list of libs to punt,
116 rather then just trying to slide the lib into existing CONTENTS.
117
118
119 > > > > Tracking BINCOMPAT should *not* be that hard.
120 > > >
121 > > > it's one more thing to keep track of and considering all of the
122 > > > possibilities i outlined, a single maintainer can easily lose his sanity
123 > > > ... or you force wasted rebuilds on people when it's only needed in some
124 > > > circumstances
125 > >
126 > > How exactly is this forcing wasted rebuilds? You're stating that
127 > > maintainers are going to bump it willy nilly. As I said, it is a key
128 > > that would be bumped *as needed*, and would only affected pkgs that
129 > > specified that node as a binding dep (specially marked atom).
130 >
131 > no, i mean for example scenarios where a package provides more than one
132 > library (say libfoo and libbar) and only one of those changes ABI values (say
133 > libfoo.0 -> libfoo.1). if another package links against just libbar, you've
134 > got a pointless rebuild.
135
136 If one changes it's version, it's a fair bet that any consumer of that
137 pkg is linked to more then just that lib; as such they would be
138 rebuilt *anyways*.
139
140
141 > > Seriously, maintainers ought to know *now* when they're forcing a
142 > > revdep on folks systems, I'm not seeing the massive overhead from
143 > > pushing that info into a var, nor am I seeing mass forced useless
144 > > rebuilds.
145 >
146 > there's a ton of things maintainers ought to know ... pretty soon our package
147 > maintainers are going to have to be gods if they want to write an ebuild as
148 > they're going to have to know every detail about the package inside and out
149
150 Sorry, but if a developer is bumping a pkg and doesn't even look to
151 see if the damn thing is potentially going to break others via soname
152 changes, that maintainer is being an idiot.
153
154 Being aware of a high level detail like "hey, my package installs a
155 library, and they changed the soname" isn't being god like, it's
156 being at least haphazardly concerned about QA of the tree.
157
158 Worth noting this exact scenario is already laid out in the quizes
159 too; 'sposed to know the affects of stabilizing a library.
160
161
162 > > Realistically, just the same as the NEEDED solution has holes, the
163 > > maintaining dev can screw up and miss a BINCOMPAT bump. Difference is
164 > > that they can do something for BINCOMPAT; hell, QA checks can be
165 > > automated to catch missing BINCOMPAT bumps.
166 >
167 > what's the difference ? in my scenario they dont have to do anything because
168 > the bump would have been caught automatically ?
169
170 Your solution has holes up the ying yang for actual
171 implementation/handling of it due to the fact it's forcing the
172 resolver to be an effective recalc at each step, and relies on duct
173 taping metadata handling to try and exempt bits of lingering data.
174
175 You move that data up front, the implementation is actually sane, it's
176 deterministic (thus *representative* to the user) and is more
177 powerful.
178
179
180 > > KEYWORDed arches bit, bit unlikely that the underlying arch is
181 > > actually going to screw with the soname, thus I'd want actual examples
182 > > backing it up.
183 >
184 > how about libc.so from glibc and libgcc.so from gcc ? or would you like other
185 > packages ?
186
187 Considering such a change would be internal ABI, NEEDED doesn't
188 actually fix anything; if it were a soname change, level playing
189 ground again.
190
191 Admittedly, BINCOMPAT breaks down there due to not being fine grained
192 enough if it's internal changes; that however doesn't make NEEDED a
193 good solution, just means that my alternative has a hole.
194
195
196 > > Besides, again, for keywording, the dev *should* be compiling it, so
197 > > there is a way to catch it :P. A revbump isn't going to break things
198 > > unless the dev is introducing something intrusive, minor version bumps
199 > > (1.1.0 to 1.1.1) shouldn't be again, introducing anything intrusive;
200 > > regardless, dev should know the high level affects of their change
201 >
202 > uhh, there is no such requirement at this time for revbumping on different
203 > arches ... currently we say the maintainer tests for his arches and leaves
204 > all the others as ~arch
205
206 Reiterating; devs should know the high level affects of their changes.
207
208 If it's going to change the soname version, they should know from the
209 get go- unless it's an arch specific breakage (which I still posit is
210 the rare/corner case), they will *see* it for their arch and bump it
211 already.
212
213 They keywording comment above is specific to moving from ~arch to arch
214 also; for new bumps, the ~arch keywords carry forward, but see the
215 paragraph above re: it.
216
217
218 > > Inducing a revdep-rebuild is definitely one of those high level
219 > > changes they should be aware of *prior* to the bump.
220 >
221 > and yet we look at our history of so many packages being bumped with ABI
222 > changes and users having broken-as-shit systems ... i'm accounting for the
223 > worst; not hoping for the best
224
225 ABI changes aren't fixed by NEEDED. Presume you meant soname changes.
226
227 Stating that things are broken doesn't make NEEDED automagically
228 better either; *both* enable a way to fix it, so you need to justify
229 the "accounting for the worst; not hoping for the best".
230
231
232 > > > or dig
233 > > > through the source code line by line and hope to catch all such cases by
234 > > > hand/eye ?
235 > >
236 > > Bit of FUD here, although I spose I'll just point out that if so's
237 > > change as massively as you're implying above, the affects on -p are
238 > > that much worse.
239 >
240 > awesome, mark something you disagree with as FUD, problem solved. my point is
241 > that you can never know completely for sure the behavior of a package without
242 > digging through the entire source code.
243
244 It's FUD due to the fact NEEDED suffers the same fucking issue. The
245 irony is that BINCOMPAT would at least give a knob to mark it as a
246 breakage, NEEDED cannot.
247
248
249 > > bind the checks in as a QA measure, enabled via FEATURES=stricter,
250 > > used to maintain a metadata var.
251 >
252 > a lot of things fail already with FEATURES=stricter ... too bad we dont have
253 > per-package env var support as then maintainers could just flag all their own
254 > packages as stricter without wanting to shoot themselves due to everyone
255 > else's package failures
256
257 Take it up with your fellow portage devs then. It was shot down
258 internally multiple times due to the affects it has on the python
259 side.
260
261 If you want to get into per package features, split a thread and I'll
262 let marius argue over it.
263
264
265 > > Literally, pkg x version y forced a rebuild. revdep is annoying, but
266 > > stats would be useful to actually evaluate the seperate proposals;
267 > > related, getting stats for how often the various updaters were
268 > > required to be ran for a particular pkgs upgrade would be useful in
269 > > evaluating that particular gap NEEDED has.
270 >
271 > any openssl version bump where the numerical value changes ... only the letter
272 > updates preserve ABI compat (0.9.7[a-z] are compat)
273 >
274 > expat-1.x -> expat-2.x
275 >
276 > iirc, tcl-8.3.x -> tcl-8.4.x
277 >
278 > readline-4.x -> readline-5.x
279 >
280 > ncurses-4.x -> ncurses-5.x
281
282 Was looking for stats comparing breakage to a subset of the tree; iow,
283 out of N packages, how many actually induce a rebuild? Yes it's
284 slightly nasty generating those stats (tinderbox?), but it would be
285 useful.
286
287
288 Frankly, if you don't believe me or think my points are correct about
289 how a NEEDED based solution is going to suck, zac/jason/genone/anyone
290 else. They're going to point out the same flaws.
291
292 ~harring

Replies

Subject Author
Re: [gentoo-dev] RFC about another *DEPEND variable Mike Frysinger <vapier@g.o>