1 |
On Sat, Sep 30, 2006 at 02:01:08PM -0400, Mike Frysinger wrote: |
2 |
> On Wednesday 27 September 2006 03:54, Brian Harring wrote: |
3 |
> > > > Bleh, this is getting back to exactly my point that it's unbounded |
4 |
> > > > resolution. To support this, every step of execution would require |
5 |
> > > > scanning for dangling nodes to punt; aside from invalidating -p's |
6 |
> > > > output it *still* is a collision-protect hit. |
7 |
> > > |
8 |
> > > when the package upgrade detects an ABI change you recalculate the |
9 |
> > > packages that need to be rebuilt ... |
10 |
> > |
11 |
> > Reiterating, this screws over any form of up front calculation; |
12 |
> > dialup users/per minute connections can't rely on emerge -f to |
13 |
> > actually fetch all involved sources, -p results ain't guranteed |
14 |
> > valid, parallelization must now lock at each threads merge on the off |
15 |
> > chance a recalc is forced. |
16 |
> |
17 |
> it does indeed prevent full up front calculation. we can always make the |
18 |
> behavior configurable; revdep on the fly or allow people to break it up. the |
19 |
> key being that their system will still continue to function as the ABI lib |
20 |
> has been preserved automagically |
21 |
|
22 |
Folks aren't that daft; you make it an optional component, it means a |
23 |
*proper* solution will never be pushed because the duct tape is in |
24 |
place already. |
25 |
|
26 |
If that's what folks want, sure, but what you're proposing is just |
27 |
sliding NEEDED in as the defacto solution without labeling it as such. |
28 |
|
29 |
|
30 |
> > > dangling nodes get recorded in the new package and considering these old |
31 |
> > > files are not detrimental to the health of the system, you can do such a |
32 |
> > > cleanup once at the end of the emerge |
33 |
> > |
34 |
> > It's not orphaning files, but your scheme doesn't work for any form of |
35 |
> > interruption; failed builds, user intervention, etc, they all can |
36 |
> > leave the lib stuck in the new contents. |
37 |
> |
38 |
> huh ? i outlined in a previous e-mail how by simply ordering the operations |
39 |
> sanely, there is no race condition |
40 |
|
41 |
Re-read your emails, and mine please. The scenario I pointed out was |
42 |
ctrl+c'ing the merge while it's doing a rebuild for lib1 to lib2. |
43 |
|
44 |
Now how does portage know that it needs to complete that upgrade for |
45 |
the next emerge action? How does it know to even scan for lib1, let |
46 |
alone punting? |
47 |
|
48 |
It's *not* simple, and I keep pointing out this issue (and you're |
49 |
missing it every damn time). Please address it, or point to where you |
50 |
have (gone over the thread and still not seeing anything even |
51 |
remotely touching this flaw). |
52 |
|
53 |
|
54 |
> > Dealing with that possibility means the manager has to maintain on |
55 |
> > disk a list of the refcount of each dangling libs to decrement, |
56 |
> > unmerge has to modify said list, etc. |
57 |
> |
58 |
> which is already being done in the NEEDED file ... funny how unpainless it is |
59 |
> to generate that file |
60 |
|
61 |
First of all, unpainless is painful. Which is apt, actually. |
62 |
|
63 |
Familiar with old style virtuals? That whole, "you have to walk the |
64 |
whole vdb to collect all old style virtuals" ? |
65 |
|
66 |
This is the same damn thing. |
67 |
|
68 |
There is no refcount maintained via NEEDED, just a list of libs a |
69 |
binary uses; you _still_ have to walk the damn vdb to build the |
70 |
refcount list. |
71 |
|
72 |
Now either you're forcing a fairly huge mapping in memory, or you're |
73 |
forcing a scan of the vdb for every pkg operation that has NEEDED |
74 |
entries it will install. |
75 |
|
76 |
So no, NEEDED doesn't cover this, and my point still stands. |
77 |
|
78 |
|
79 |
> > > > It also involves changing vdb nodes from "installed and usable" to |
80 |
> > > > "installed/usable" or "lingering" |
81 |
> > > |
82 |
> > > no ... the old versions get merged into the new one as their existence is |
83 |
> > > purely hidden |
84 |
> > |
85 |
> > How exactly are they going to be hidden? A new file? If so, |
86 |
> > backwards compatibility for vdb for transitioning in such a support |
87 |
> > has to be addressed. |
88 |
> |
89 |
> purely hidden from the standpoint of any new package trying to use it ... |
90 |
> since you're only installing the runtime ABI library, you cannot link against |
91 |
> it or utilize it any way other than existing files. |
92 |
|
93 |
Except for dlopen, but thats again besides my original point; how do |
94 |
you note to portage that the lib is one to watch so it can be punted? |
95 |
|
96 |
You're suggesting it get shoved into contents, and for portage to |
97 |
identify it, it has to do a revdep mapping on the fly to find it. |
98 |
|
99 |
This *sucks* massively, both from a speed and complexity standpoint; |
100 |
further, the lib isn't hidden from pkg ops (say quickpkging, or |
101 |
binpkging) in any way so the cruft spreads. That's surprisingly minor |
102 |
in comparison to implications of relying on refcounting to identify |
103 |
the lib to punt. |
104 |
|
105 |
If the lib to punt isn't tracked in some fashion, the only algo |
106 |
you're left with is attempting to find all libs that have a refcount |
107 |
of zero, and punt those- obviously, this is going to screw up just |
108 |
about every single dlopen based linkage, literally, suck an algo |
109 |
won't spot that python dlopen's it's modules, and would think the |
110 |
refcount was zero (thus the so could get booted). |
111 |
|
112 |
The response there is to add blacklists, directories to *not* inspect, |
113 |
which is a further hack to try and make this tank fly. |
114 |
|
115 |
*OR*, you're stuck maintaining a seperated list of libs to punt, |
116 |
rather then just trying to slide the lib into existing CONTENTS. |
117 |
|
118 |
|
119 |
> > > > Tracking BINCOMPAT should *not* be that hard. |
120 |
> > > |
121 |
> > > it's one more thing to keep track of and considering all of the |
122 |
> > > possibilities i outlined, a single maintainer can easily lose his sanity |
123 |
> > > ... or you force wasted rebuilds on people when it's only needed in some |
124 |
> > > circumstances |
125 |
> > |
126 |
> > How exactly is this forcing wasted rebuilds? You're stating that |
127 |
> > maintainers are going to bump it willy nilly. As I said, it is a key |
128 |
> > that would be bumped *as needed*, and would only affected pkgs that |
129 |
> > specified that node as a binding dep (specially marked atom). |
130 |
> |
131 |
> no, i mean for example scenarios where a package provides more than one |
132 |
> library (say libfoo and libbar) and only one of those changes ABI values (say |
133 |
> libfoo.0 -> libfoo.1). if another package links against just libbar, you've |
134 |
> got a pointless rebuild. |
135 |
|
136 |
If one changes it's version, it's a fair bet that any consumer of that |
137 |
pkg is linked to more then just that lib; as such they would be |
138 |
rebuilt *anyways*. |
139 |
|
140 |
|
141 |
> > Seriously, maintainers ought to know *now* when they're forcing a |
142 |
> > revdep on folks systems, I'm not seeing the massive overhead from |
143 |
> > pushing that info into a var, nor am I seeing mass forced useless |
144 |
> > rebuilds. |
145 |
> |
146 |
> there's a ton of things maintainers ought to know ... pretty soon our package |
147 |
> maintainers are going to have to be gods if they want to write an ebuild as |
148 |
> they're going to have to know every detail about the package inside and out |
149 |
|
150 |
Sorry, but if a developer is bumping a pkg and doesn't even look to |
151 |
see if the damn thing is potentially going to break others via soname |
152 |
changes, that maintainer is being an idiot. |
153 |
|
154 |
Being aware of a high level detail like "hey, my package installs a |
155 |
library, and they changed the soname" isn't being god like, it's |
156 |
being at least haphazardly concerned about QA of the tree. |
157 |
|
158 |
Worth noting this exact scenario is already laid out in the quizes |
159 |
too; 'sposed to know the affects of stabilizing a library. |
160 |
|
161 |
|
162 |
> > Realistically, just the same as the NEEDED solution has holes, the |
163 |
> > maintaining dev can screw up and miss a BINCOMPAT bump. Difference is |
164 |
> > that they can do something for BINCOMPAT; hell, QA checks can be |
165 |
> > automated to catch missing BINCOMPAT bumps. |
166 |
> |
167 |
> what's the difference ? in my scenario they dont have to do anything because |
168 |
> the bump would have been caught automatically ? |
169 |
|
170 |
Your solution has holes up the ying yang for actual |
171 |
implementation/handling of it due to the fact it's forcing the |
172 |
resolver to be an effective recalc at each step, and relies on duct |
173 |
taping metadata handling to try and exempt bits of lingering data. |
174 |
|
175 |
You move that data up front, the implementation is actually sane, it's |
176 |
deterministic (thus *representative* to the user) and is more |
177 |
powerful. |
178 |
|
179 |
|
180 |
> > KEYWORDed arches bit, bit unlikely that the underlying arch is |
181 |
> > actually going to screw with the soname, thus I'd want actual examples |
182 |
> > backing it up. |
183 |
> |
184 |
> how about libc.so from glibc and libgcc.so from gcc ? or would you like other |
185 |
> packages ? |
186 |
|
187 |
Considering such a change would be internal ABI, NEEDED doesn't |
188 |
actually fix anything; if it were a soname change, level playing |
189 |
ground again. |
190 |
|
191 |
Admittedly, BINCOMPAT breaks down there due to not being fine grained |
192 |
enough if it's internal changes; that however doesn't make NEEDED a |
193 |
good solution, just means that my alternative has a hole. |
194 |
|
195 |
|
196 |
> > Besides, again, for keywording, the dev *should* be compiling it, so |
197 |
> > there is a way to catch it :P. A revbump isn't going to break things |
198 |
> > unless the dev is introducing something intrusive, minor version bumps |
199 |
> > (1.1.0 to 1.1.1) shouldn't be again, introducing anything intrusive; |
200 |
> > regardless, dev should know the high level affects of their change |
201 |
> |
202 |
> uhh, there is no such requirement at this time for revbumping on different |
203 |
> arches ... currently we say the maintainer tests for his arches and leaves |
204 |
> all the others as ~arch |
205 |
|
206 |
Reiterating; devs should know the high level affects of their changes. |
207 |
|
208 |
If it's going to change the soname version, they should know from the |
209 |
get go- unless it's an arch specific breakage (which I still posit is |
210 |
the rare/corner case), they will *see* it for their arch and bump it |
211 |
already. |
212 |
|
213 |
They keywording comment above is specific to moving from ~arch to arch |
214 |
also; for new bumps, the ~arch keywords carry forward, but see the |
215 |
paragraph above re: it. |
216 |
|
217 |
|
218 |
> > Inducing a revdep-rebuild is definitely one of those high level |
219 |
> > changes they should be aware of *prior* to the bump. |
220 |
> |
221 |
> and yet we look at our history of so many packages being bumped with ABI |
222 |
> changes and users having broken-as-shit systems ... i'm accounting for the |
223 |
> worst; not hoping for the best |
224 |
|
225 |
ABI changes aren't fixed by NEEDED. Presume you meant soname changes. |
226 |
|
227 |
Stating that things are broken doesn't make NEEDED automagically |
228 |
better either; *both* enable a way to fix it, so you need to justify |
229 |
the "accounting for the worst; not hoping for the best". |
230 |
|
231 |
|
232 |
> > > or dig |
233 |
> > > through the source code line by line and hope to catch all such cases by |
234 |
> > > hand/eye ? |
235 |
> > |
236 |
> > Bit of FUD here, although I spose I'll just point out that if so's |
237 |
> > change as massively as you're implying above, the affects on -p are |
238 |
> > that much worse. |
239 |
> |
240 |
> awesome, mark something you disagree with as FUD, problem solved. my point is |
241 |
> that you can never know completely for sure the behavior of a package without |
242 |
> digging through the entire source code. |
243 |
|
244 |
It's FUD due to the fact NEEDED suffers the same fucking issue. The |
245 |
irony is that BINCOMPAT would at least give a knob to mark it as a |
246 |
breakage, NEEDED cannot. |
247 |
|
248 |
|
249 |
> > bind the checks in as a QA measure, enabled via FEATURES=stricter, |
250 |
> > used to maintain a metadata var. |
251 |
> |
252 |
> a lot of things fail already with FEATURES=stricter ... too bad we dont have |
253 |
> per-package env var support as then maintainers could just flag all their own |
254 |
> packages as stricter without wanting to shoot themselves due to everyone |
255 |
> else's package failures |
256 |
|
257 |
Take it up with your fellow portage devs then. It was shot down |
258 |
internally multiple times due to the affects it has on the python |
259 |
side. |
260 |
|
261 |
If you want to get into per package features, split a thread and I'll |
262 |
let marius argue over it. |
263 |
|
264 |
|
265 |
> > Literally, pkg x version y forced a rebuild. revdep is annoying, but |
266 |
> > stats would be useful to actually evaluate the seperate proposals; |
267 |
> > related, getting stats for how often the various updaters were |
268 |
> > required to be ran for a particular pkgs upgrade would be useful in |
269 |
> > evaluating that particular gap NEEDED has. |
270 |
> |
271 |
> any openssl version bump where the numerical value changes ... only the letter |
272 |
> updates preserve ABI compat (0.9.7[a-z] are compat) |
273 |
> |
274 |
> expat-1.x -> expat-2.x |
275 |
> |
276 |
> iirc, tcl-8.3.x -> tcl-8.4.x |
277 |
> |
278 |
> readline-4.x -> readline-5.x |
279 |
> |
280 |
> ncurses-4.x -> ncurses-5.x |
281 |
|
282 |
Was looking for stats comparing breakage to a subset of the tree; iow, |
283 |
out of N packages, how many actually induce a rebuild? Yes it's |
284 |
slightly nasty generating those stats (tinderbox?), but it would be |
285 |
useful. |
286 |
|
287 |
|
288 |
Frankly, if you don't believe me or think my points are correct about |
289 |
how a NEEDED based solution is going to suck, zac/jason/genone/anyone |
290 |
else. They're going to point out the same flaws. |
291 |
|
292 |
~harring |