1 |
On Fri, Mar 27, 2020 at 7:00 AM <michael.lienhardt@×××××××.net> wrote: |
2 |
|
3 |
> |
4 |
> ----- Alec Warner <antarus@g.o> a écrit : |
5 |
> > On Tue, Mar 24, 2020 at 11:31 AM <michael.lienhardt@×××××××.net> wrote: |
6 |
> > > However, I still doubt that only storing the soname dependencies is |
7 |
> enough. |
8 |
> > > Consider package A (that cannot be recompiled) that depends on package |
9 |
> B |
10 |
> > > which provides lib L.so. |
11 |
> > > B is recompiled with different use flags, which put different |
12 |
> > > functionalities in L.so. |
13 |
> > > The dependencies of A are still satisfied (B is installed, L.so is |
14 |
> > > available), but since the content of L.so changed, A cannot execute |
15 |
> anymore. |
16 |
> > > Hypothetically, can this scenario occur? |
17 |
> > > Can this scenario occur in practice? |
18 |
> > > Is there a way in emerge/portage to avoid it? |
19 |
> |
20 |
> |
21 |
> > > You have far more experience than me on this, and it would be nice for |
22 |
> me |
23 |
> > > to know what I'm up against. |
24 |
> > |
25 |
> > A lot of this has to do with the specifics of how package managers manage |
26 |
> > system state, as well as various quirks of subsets of the tree. For |
27 |
> > example, a perl upgrade (X->Y) will often break perl modules who expect |
28 |
> > perl-X, but get perl-Y. So one fix is to try to keep perl-X installed (so |
29 |
> > we SLOT perl and have N perls installed.) Then you need to decide which |
30 |
> > version of perl to build things against (X or Y, or both?) We took this |
31 |
> > tactic in the python ecosystem; but perl is not slotted in Gentoo, and so |
32 |
> > upgrading perl breaks all perl modules. There is a tool |
33 |
> > (gentoo-perl-cleaner) that will walk the deptree and fix all of these |
34 |
> > broken packages that you run after an upgrade. |
35 |
> > |
36 |
> > I'm not sure it's strictly avoidable. You could build perl-Y, then |
37 |
> rebuild |
38 |
> > all perl-modules against perl-Y, then merge the entire result to the |
39 |
> > livefs. This will reduce the breakage time but likely not eliminate it; |
40 |
> > plus it seems hard to implement in practice without modern filesystem |
41 |
> tools |
42 |
> > (overlayfs, btrfs, zfs or similar tech to make it atomic.) It also |
43 |
> doesn't |
44 |
> > account for executing code. What happens to perl-X code that is executing |
45 |
> > when you unmerge perl-X? The short answer is that code might break and |
46 |
> > 'proper' management means you should restart services after an upgrade |
47 |
> > (something Gentoo doesn't typically do; but is common in Debian for |
48 |
> > example.) |
49 |
> |
50 |
> Many thanks for this answer. |
51 |
> To sum up what I understood, the problem is not really the dependencies, |
52 |
> but which recompilation (and service restart) are triggered with an update. |
53 |
> |
54 |
|
55 |
Yes and no. Assume a spherical package manager that could detect |
56 |
everything, we basically need to do the following for a perl X -> Y upgrade. |
57 |
|
58 |
1 User triggers X - Y upgrade. |
59 |
2 build perl-Y, but don't merge it to the livefs. |
60 |
3 numerate all deps of perl-Y, build those (but don't merge them to the |
61 |
livefs, but they need to build against perl-Y, not perl-X) |
62 |
4 with atomic_transaction: |
63 |
4a Merge perl-Y to the livefs |
64 |
4b Merge perl-Y's dependencies to the live fs |
65 |
5 restart anything that is running perl-X, so that it runs with perl-Y |
66 |
6 unmerge perl-X |
67 |
|
68 |
In practice we cannot always do 3 or 4 or 5. We will miss dependencies (due |
69 |
to missing depgraph information) both in the package depgraph as well as |
70 |
the service depgraph. |
71 |
|
72 |
So in practice we do: |
73 |
1 user triggers X -Y upgrade. |
74 |
2 build perl-Y, merge it to the livefs, unmerge perl-X |
75 |
3 run gentoo-perl-cleaner to upgrade the dependencies broken by the X - Y |
76 |
upgrade (via slot deps, or whatever mechanism, it can vary.) |
77 |
4 restart anything that is running perl-X |
78 |
|
79 |
And just accept that from 2-4..well stuff will be broken. We can minimize |
80 |
the time by building binpkgs ahead of time for example, and merging the |
81 |
binpkgs in a parallel. Note that 5 and 4 are the same problem in both |
82 |
lists. Note that per the guide linked below, sometimes 2-3 can be done 'at |
83 |
once', although again practically speaking "During Perl upgrade, packages |
84 |
that depend on Perl may become unavailable." even if they are all handled |
85 |
by emerge, because there are many race conditions in the order that |
86 |
packages get merged to the livefs. |
87 |
|
88 |
|
89 |
> In gentoo, there is the ":=" slot operator (and others similar) in |
90 |
> dependencies that trigger the recompilation when the dependency's slot |
91 |
> change, but this is the only existing mechanism. |
92 |
> And this is why every time perl changes, the compilation of its modules is |
93 |
> not triggered and they are most probably broken. |
94 |
> Correct? |
95 |
> |
96 |
|
97 |
They are supposed to be triggered: |
98 |
https://wiki.gentoo.org/wiki/Perl#Upgrading_.28major_version.29 for example |
99 |
says this. But this is up to how callers run the tool. For example gentoo |
100 |
infra executes emerge via puppet, and we never execute emerge -uDNav |
101 |
--with-bdeps=y --backtrack=100 --autounmask-keep-masks=y @world, we have |
102 |
our own piece of code that handles perl upgrades. |
103 |
|
104 |
|
105 |
> And then, in this context, keeping the installed packages' dependencies |
106 |
> consistent is up to debate: packages will get broken in any case... |
107 |
> It is clearly impossible to have a tool that automatically detect all |
108 |
> implementation dependency breakage. |
109 |
> |
110 |
> Again, there's something I probably don't see: why was slot operators |
111 |
> chosen (among other possibilities) as a mechanism to trigger recompilation? |
112 |
> I'm very grateful to you all for the time you take to read and answer my |
113 |
> questions. |
114 |
|
115 |
|
116 |
Sorry I'm being overly academic. My concern earlier is that you mentioned a |
117 |
goal of "never breaking installed packages' which I found to be a fairly |
118 |
audacious goal. The idea is that we should build tools that achieve this |
119 |
practically (e.g. via heuristics such as := slot operators) while |
120 |
understanding that the complexities of application deploys are legion and |
121 |
the tool will never handle them all. So the goal of never breaking them is |
122 |
more an idealistic goal rather than a practical reality. |
123 |
|
124 |
-A |
125 |
|
126 |
|
127 |
|
128 |
> Best, |
129 |
> Michael |
130 |
> |
131 |
> |