Gentoo Archives: gentoo-project

From: "Michał Górny" <mgorny@g.o>
To: Michael Palimaka <kensington@g.o>
Cc: gentoo-project@l.g.o
Subject: Re: [gentoo-project] Re: Call for agenda items - Council meeting 2014-08-12
Date: Fri, 01 Aug 2014 19:47:44
Message-Id: 20140801214751.76a4e8d9@pomiot.lan
In Reply to: [gentoo-project] Re: Call for agenda items - Council meeting 2014-08-12 by Michael Palimaka
1 Dnia 2014-08-01, o godz. 05:52:40
2 Michael Palimaka <kensington@g.o> napisał(a):
4 > On 08/01/2014 05:12 AM, Michał Górny wrote:
5 >
6 > >> If we are to change our default dependency model, we need to do it
7 > >> properly - we need wider consensus, more open discussion of what's
8 > >> happening, and a proper plan of how to handle the pitfalls of the new
9 > >> model. Otherwise, we're just trading one set of problems for another.
10 > >
11 > > Yes, exactly. We need to get dynamic-deps right if they are ever
12 > > supposed to become the default. That's one of the reasons that we want
13 > > to revert the problematic changes and make Portage use the default
14 > > model once again.
15 > What do you mean? Dynamic dependencies are the current default by
16 > definition.
18 Unless I've missed something big, dynamic dependencies were enabled
19 in Portage without prior discussion or much testing. The developers
20 started to rely on it though that was never desired or agreed on. As
21 far as I'm aware, the Council never voted on the matter but it rather
22 spread from developer to developer. This also implies that it was never
23 properly described, and most of the developers do not know the details
24 or the pitfalls.
26 That said, I'd dare say that the Gentoo contract still obliges us to
27 follow PMS, and therefore the dependency model implied by PMS
28 is the default one.
30 > >> Specifically, I request the Council block the removal of dynamic
31 > >> dependencies until the following issues are resolved:
32 > >>
33 > >> 1. Although there has been considerable discussion[2] regarding dynamic
34 > >> dependencies in general, there has been no specific discussion regarding
35 > >> their actual removal.
36 > >
37 > > The thread pretty quickly turned into a bikeshed about that, so I don't
38 > > understand what you're talking about.
39 > Perhaps I missed it due to the size of the thread, but the discussion
40 > appeared to be more abstract rather than concerning any definite change.
41 > I see relatively little comment from Portage team members in any case.
42 >
43 > >> 2. The Portage team had made no announcement of their decision, nor do
44 > >> they apparently intend to until 30 days prior to a new Portage release.
45 > >> This is not adequate time for such a substantial change.
46 > >
47 > > I don't know what you mean by 'they apparently intend' but that sounds
48 > > offensive. I'd really prefer if you sticked to the facts.
49 > How is that offensive? It is the apparent intention I've been able to
50 > discern from the limited information the Portage team is providing.
51 >
52 > > The Portage team is going to announce the decision when it's final
53 > > and the code necessary for non-painful migration is in place.
54 > > Otherwise, the announcement would only bring barren discussion that
55 > > couldn't be supported by facts or numbers.
56 > This is not acceptable for such a significant change.
58 I think we misunderstand each other.
60 The Portage team is quite small, and has a lot of work to do. As I
61 mentioned, we are still collecting data and focusing on having the code
62 necessary to collect it. In particular, today I've submitted the last
63 patch necessary for @changed-deps set to work properly.
65 By using this set, we can check how many packages had dependencies
66 updated without revision bump and therefore estimate the scale
67 of the problem. Furthermore, if dynamic dependencies are disabled
68 in Portage, it will help users fixing the inconsistencies in their vdb
69 which may prevent emerge from working properly afterwards.
71 I simply meant that we haven't announced anything yet because we're
72 nowhere close to doing this. We don't want to bother people too much
73 before we know the scale of the problem, and before we can provide real
74 data and suggestions.
76 > >> 3. Few of the Portage team members were present[3] for the meeting, and
77 > >> no vote was held to reach the decision.
78 > >
79 > > I don't understand how this is relevant.
80 > It's not acceptable for one or two people to make a decision that
81 > affects the entire distribution, let alone if they don't have the
82 > consensus of their own team.
84 While I don't want to diminish the role of the remaining team members,
85 I would like to point out that those two were the team leads
86 and the most active members. And those members didn't reach a strong
87 conclusion -- as you pointed out, they never even voted. We just agreed
88 on a goal we're going to work on reaching.
90 > >> 4. While there is a good description of the theoretical issues affecting
91 > >> both dependency models[4], multiple requests for specific examples of
92 > >> in-tree breakage have been ignored. This makes it difficult to assess
93 > >> the actual breakage that removing dynamic dependencies is supposed to
94 > >> address.
95 > >
96 > > The listing of theoretical issues is wrong and doesn't correspond to
97 > > Portage code, and yes, that's my fault. However, it only proves that
98 > > nobody on the thread bothered to look at the relevant Portage code,
99 > > even the people who started providing patches...
100 > The burden of proof is on the person wanting to make the change, so it
101 > would be nice if the Portage team would provide better information.
102 > Despite repeated requests, we have little information to substantiate
103 > claims like "it's fundamentally broken" and "dynamic dependencies is a
104 > bug. We decided to fix this regression."
106 There are basically two sides to the issue: the issues with dynamic
107 dependencies themselves and the issues caused by developers relying
108 on them. An elaborate listing would likely require much more skill than
109 I have.
111 However, a short listing would include:
113 1. the dependency tree for installed packages is non-deterministic. It
114 can suffer rapid changes due to seemingly irrelevant events, most
115 importantly -- inability to find the original ebuild. This involves not
116 only removal of ebuilds but also temporary changes to PORTDIR_OVERLAY,
117 for example. Users don't expect that and the issues are very hard to
118 debug -- why is A pulling in non-existing package B? it's nowhere in
119 A's deps.
121 2. dynamic dependencies allow for dependencies stored in vdb to become
122 inconsistent. I mean, since portage usually doesn't use them
123 and the developers change dependencies in-place, the vdb dependency
124 tree may contain completely impossible combinations of packages
125 (depending on the time a particular package was installed). The problem
126 is that only emerge does this, and the vdb is simply broken to any
127 other package manager or tool -- including portage-utils (AFAICS
128 'qdepends -Q' gives meaningless output nowadays), eix, gentoolkit.
130 3. most of the developers don't understand when the revbump becomes
131 necessary because of dynamic-deps (i.e. when you don't want them to
132 apply). This occurs pretty rarely yet it's something I doubt we will
133 ever be able to change. It is simply too confusing.
135 4. getting := and dynamic-deps working together is pretty hard,
136 and the code doing that in Portage is not very good. I suspect that
137 the resulting dependency trees may be the cause of some issues people
138 are reporting. If backtracking is involved, dynamic-deps are also
139 likely to cause much longer dependency resolution. However, it's all
140 very complex and hard to debug, so I have no definitive proof.
142 I believe that fixing many of the issues in Portage will require severe
143 changes to the code. Disabling the dynamic dependency support will
144 likely make debugging easier for us, and therefore get us closer to
145 having properly working dependency resolver.
147 Additionally, disabled dynamic dependencies give users the ability to
148 choose any package manager for Gentoo. With dynamic dependencies
149 enabled, emerge breaks vdb pretty soon (as pointed out in (2)) which in
150 turn breaks other package managers, and leaves user with no choice but
151 to deepen the breakage.
153 > >> 5. The removal plan doesn't consider the impact from increased number of
154 > >> rebuilds due to revbumps containing only dependency changes. Without
155 > >> replacement functionality, widespread virtual or slot changes can cause
156 > >> hundreds of packages to be rebuilt.
157 > >
158 > > Please support this claim with specific examples or numbers. Otherwise,
159 > > it's just a 'theoretical issue' that you can't support.
160 > I'm happy to provide more examples beyond the 77
161 > useless-but-apparently-acceptable rebuilds you mentioned below if
162 > necessary. It's not hard to check yourself, though. Almost 9000 ebuilds
163 > depend on virtual/pkgconfig. Even if we assume that any future virtual
164 > introduction will only affect an absolute maximum of 10% of that number
165 > of packages, that's still a ridiculous number of unnecessary rebuilds.
167 If you assume that the immediate rebuild is actually a necessity.
168 Please note that many of the changes can actually be carried out within
169 a reasonable timeframe as part of package upgrade.
171 Given your example, the change was necessary to support a different
172 implementation of pkg-config. We may argue that choosing a non-default
173 implementation is pretty rare, and requesting those people to rebuild
174 a number of packages shouldn't really be a problem.
176 > > There is a lot of FUD about unnecessary rebuilds. Sadly, most people
177 > > seem to fight a holy war against them without realizing the real
178 > > impact. In fact, more unnecessary rebuilds are caused by unnecessary
179 > > USE flags than by dependency changes.
180 > Do you have some numbers to back up that statement? I can think of very
181 > few instances when I have had to rebuild against my will due to USE flag
182 > changes.
184 This is hard to count since it differs on per-case basis. I'm talking
185 about the two common cases here: when user doesn't know/notice that he
186 has to enable/disable a particular flag, and when user installs a new
187 package that requires USE changes on other packages.
189 > > Yet the same people believe in
190 > > adding more flags to contain even more minor aspects of packages...
191 > Are you referring to flags for things like logrotate or systemd units?
192 > When they were around, I either had them enabled or I didn't - I was
193 > never forced to rebuild because of them.
195 Rather flags for various package install aspects -- switching very
196 commonly used aspects, changing library install details. Think of
197 USE=tinfo on ncurses or attempt to add a flag to control library
198 install to pypy. But that's really irrelevant to the topic.
200 --
201 Best regards,
202 Michał Górny


File name MIME type
signature.asc application/pgp-signature