Re: [gentoo-project] Re: Call for agenda items - Council meeting 2014-08-12

[Michał Górny]

Dnia 2014-08-01, o godz. 05:52:40
Michael Palimaka <kensington@g.o> napisał(a):

> On 08/01/2014 05:12 AM, Michał Górny wrote:
>
> >> If we are to change our default dependency model, we need to do it
> >> properly - we need wider consensus, more open discussion of what's
> >> happening, and a proper plan of how to handle the pitfalls of the new
> >> model. Otherwise, we're just trading one set of problems for another.
> >
> > Yes, exactly. We need to get dynamic-deps right if they are ever
> > supposed to become the default. That's one of the reasons that we want
> > to revert the problematic changes and make Portage use the default
> > model once again.
> What do you mean? Dynamic dependencies are the current default by
> definition.

Unless I've missed something big, dynamic dependencies were enabled
in Portage without prior discussion or much testing. The developers
started to rely on it though that was never desired or agreed on. As
far as I'm aware, the Council never voted on the matter but it rather
spread from developer to developer. This also implies that it was never
properly described, and most of the developers do not know the details
or the pitfalls.

That said, I'd dare say that the Gentoo contract still obliges us to
follow PMS, and therefore the dependency model implied by PMS
is the default one.

> >> Specifically, I request the Council block the removal of dynamic
> >> dependencies until the following issues are resolved:
> >>
> >> 1. Although there has been considerable discussion[2] regarding dynamic
> >> dependencies in general, there has been no specific discussion regarding
> >> their actual removal.
> >
> > The thread pretty quickly turned into a bikeshed about that, so I don't
> > understand what you're talking about.
> Perhaps I missed it due to the size of the thread, but the discussion
> appeared to be more abstract rather than concerning any definite change.
> I see relatively little comment from Portage team members in any case.
>
> >> 2. The Portage team had made no announcement of their decision, nor do
> >> they apparently intend to until 30 days prior to a new Portage release.
> >> This is not adequate time for such a substantial change.
> >
> > I don't know what you mean by 'they apparently intend' but that sounds
> > offensive. I'd really prefer if you sticked to the facts.
> How is that offensive? It is the apparent intention I've been able to
> discern from the limited information the Portage team is providing.
>
> > The Portage team is going to announce the decision when it's final
> > and the code necessary for non-painful migration is in place.
> > Otherwise, the announcement would only bring barren discussion that
> > couldn't be supported by facts or numbers.
> This is not acceptable for such a significant change.

I think we misunderstand each other.

The Portage team is quite small, and has a lot of work to do. As I
mentioned, we are still collecting data and focusing on having the code
necessary to collect it. In particular, today I've submitted the last
patch necessary for @changed-deps set to work properly.

By using this set, we can check how many packages had dependencies
updated without revision bump and therefore estimate the scale
of the problem. Furthermore, if dynamic dependencies are disabled
in Portage, it will help users fixing the inconsistencies in their vdb
which may prevent emerge from working properly afterwards.

I simply meant that we haven't announced anything yet because we're
nowhere close to doing this. We don't want to bother people too much
before we know the scale of the problem, and before we can provide real
data and suggestions.

> >> 3. Few of the Portage team members were present[3] for the meeting, and
> >> no vote was held to reach the decision.
> >
> > I don't understand how this is relevant.
> It's not acceptable for one or two people to make a decision that
> affects the entire distribution, let alone if they don't have the
> consensus of their own team.

While I don't want to diminish the role of the remaining team members,
I would like to point out that those two were the team leads
and the most active members. And those members didn't reach a strong
conclusion -- as you pointed out, they never even voted. We just agreed
on a goal we're going to work on reaching.

> >> 4. While there is a good description of the theoretical issues affecting
> >> both dependency models[4], multiple requests for specific examples of
> >> in-tree breakage have been ignored. This makes it difficult to assess
> >> the actual breakage that removing dynamic dependencies is supposed to
> >> address.
> >
> > The listing of theoretical issues is wrong and doesn't correspond to
> > Portage code, and yes, that's my fault. However, it only proves that
> > nobody on the thread bothered to look at the relevant Portage code,
> > even the people who started providing patches...
> The burden of proof is on the person wanting to make the change, so it
> would be nice if the Portage team would provide better information.
> Despite repeated requests, we have little information to substantiate
> claims like "it's fundamentally broken" and "dynamic dependencies is a
> bug. We decided to fix this regression."

There are basically two sides to the issue: the issues with dynamic
dependencies themselves and the issues caused by developers relying
on them. An elaborate listing would likely require much more skill than
I have.

However, a short listing would include:

1. the dependency tree for installed packages is non-deterministic. It
can suffer rapid changes due to seemingly irrelevant events, most
importantly -- inability to find the original ebuild. This involves not
only removal of ebuilds but also temporary changes to PORTDIR_OVERLAY,
for example. Users don't expect that and the issues are very hard to
debug -- why is A pulling in non-existing package B? it's nowhere in
A's deps.

2. dynamic dependencies allow for dependencies stored in vdb to become
inconsistent. I mean, since portage usually doesn't use them
and the developers change dependencies in-place, the vdb dependency
tree may contain completely impossible combinations of packages
(depending on the time a particular package was installed). The problem
is that only emerge does this, and the vdb is simply broken to any
other package manager or tool -- including portage-utils (AFAICS
'qdepends -Q' gives meaningless output nowadays), eix, gentoolkit.

3. most of the developers don't understand when the revbump becomes
necessary because of dynamic-deps (i.e. when you don't want them to
apply). This occurs pretty rarely yet it's something I doubt we will
ever be able to change. It is simply too confusing.

4. getting := and dynamic-deps working together is pretty hard,
and the code doing that in Portage is not very good. I suspect that
the resulting dependency trees may be the cause of some issues people
are reporting. If backtracking is involved, dynamic-deps are also
likely to cause much longer dependency resolution. However, it's all
very complex and hard to debug, so I have no definitive proof.

I believe that fixing many of the issues in Portage will require severe
changes to the code. Disabling the dynamic dependency support will
likely make debugging easier for us, and therefore get us closer to
having properly working dependency resolver.

Additionally, disabled dynamic dependencies give users the ability to
choose any package manager for Gentoo. With dynamic dependencies
enabled, emerge breaks vdb pretty soon (as pointed out in (2)) which in
turn breaks other package managers, and leaves user with no choice but
to deepen the breakage.

> >> 5. The removal plan doesn't consider the impact from increased number of
> >> rebuilds due to revbumps containing only dependency changes. Without
> >> replacement functionality, widespread virtual or slot changes can cause
> >> hundreds of packages to be rebuilt.
> >
> > Please support this claim with specific examples or numbers. Otherwise,
> > it's just a 'theoretical issue' that you can't support.
> I'm happy to provide more examples beyond the 77
> useless-but-apparently-acceptable rebuilds you mentioned below if
> necessary. It's not hard to check yourself, though. Almost 9000 ebuilds
> depend on virtual/pkgconfig. Even if we assume that any future virtual
> introduction will only affect an absolute maximum of 10% of that number
> of packages, that's still a ridiculous number of unnecessary rebuilds.

If you assume that the immediate rebuild is actually a necessity.
Please note that many of the changes can actually be carried out within
a reasonable timeframe as part of package upgrade.

Given your example, the change was necessary to support a different
implementation of pkg-config. We may argue that choosing a non-default
implementation is pretty rare, and requesting those people to rebuild
a number of packages shouldn't really be a problem.

> > There is a lot of FUD about unnecessary rebuilds. Sadly, most people
> > seem to fight a holy war against them without realizing the real
> > impact. In fact, more unnecessary rebuilds are caused by unnecessary
> > USE flags than by dependency changes.
> Do you have some numbers to back up that statement? I can think of very
> few instances when I have had to rebuild against my will due to USE flag
> changes.

This is hard to count since it differs on per-case basis. I'm talking
about the two common cases here: when user doesn't know/notice that he
has to enable/disable a particular flag, and when user installs a new
package that requires USE changes on other packages.

> > Yet the same people believe in
> > adding more flags to contain even more minor aspects of packages...
> Are you referring to flags for things like logrotate or systemd units?
> When they were around, I either had them enabled or I didn't - I was
> never forced to rebuild because of them.

Rather flags for various package install aspects -- switching very
commonly used aspects, changing library install details. Think of
USE=tinfo on ncurses or attempt to add a flag to control library
install to pypy. But that's really irrelevant to the topic.

-- 
Best regards,
Michał Górny