1 |
Hi, everyone. |
2 |
|
3 |
TL;DR: how to deal with setuptools (and newer distutils vendored by |
4 |
setuptools) replacing .egg-info files with directories? |
5 |
|
6 |
|
7 |
I know I'm reiterating the same topic but I think we're coming to having |
8 |
to actually do something about this and I still haven't found |
9 |
a satisfactory solution. |
10 |
|
11 |
For the people new to the problem, a quick recap: Python packages |
12 |
install metadata called .egg-info (newer build systems use .dist-info |
13 |
but that's another matter). Original Python distutils installed this |
14 |
as a single file but setuptools replaced it with a directory. |
15 |
|
16 |
Now that distutils is deprecated, setuptools is vendoring its own |
17 |
version. We can switch between the system and the vendored version |
18 |
using an envvar. By default, setuptools < 60 uses system distutils, |
19 |
and >= 60 (masked) uses vendored distutils. |
20 |
|
21 |
The big problem is that switching implies changing the format, so if you |
22 |
install foo-X, then switch, then reinstall the same version of foo, |
23 |
you're going to have the file replaced by a directory. This is not |
24 |
supported by the PMS, and Portage handles it somewhat suboptimally |
25 |
(renaming the old file and leaving it orphaned). |
26 |
|
27 |
I should probably emphasize here that the .egg-info path contains |
28 |
the package version, so this is a problem only if the same upstream |
29 |
version is being reinstalled. |
30 |
|
31 |
You can easily reproduce the problem by playing with: |
32 |
|
33 |
SETUPTOOLS_USE_DISTUTILS=stdlib |
34 |
SETUPTOOLS_USE_DISTUTILS=local # vendored |
35 |
|
36 |
and repeatedly building some DISTUTILS_USE_SETUPTOOLS=no packages. |
37 |
|
38 |
|
39 |
From a quick grep, there are 179 packages using DUS=no right now |
40 |
(and there might be more using distutils without the declaration). |
41 |
I don't think we can ignore the problem. |
42 |
|
43 |
What we can do right now is force the SETUPTOOLS_USE_DISTUTILS=stdlib |
44 |
default back via the eclass or patches for the time being. This will |
45 |
let us finally unmask setuptools-60+ while delaying fixing the problem. |
46 |
However, given pypa's tendency to remove deprecated stuff quickly, this |
47 |
is unlikely to work for long. |
48 |
|
49 |
Some ideas on fixing this: |
50 |
|
51 |
1. For a start, with distutils deprecated are actually migrating to |
52 |
setuptools or other build systems upstream. This effectively solves |
53 |
the problem for us since the .egg-info switch happens on version bump |
54 |
and there is no file collision. However, this isn't going to help for |
55 |
dead projects. |
56 |
|
57 |
2. We could control the distutils version in ebuilds directly, |
58 |
i.e. force "stdlib" for the current versions and have developers switch |
59 |
to "local" on version bumps. Combined with 1., this will probably |
60 |
increase the coverage a bit but dead packages will remain in the way. |
61 |
It also relies on all devs understanding the problem. |
62 |
|
63 |
3. The developers could explicitly bump versions (i.e. create "Gentoo |
64 |
subversions") of packages that don't expect any updates. We can get |
65 |
100% coverage this way but it's hard and requires patching. |
66 |
|
67 |
4. We could have the eclasses switch to "local" model and rename |
68 |
the .egg-info files somehow at some point. The main question is "rename |
69 |
how?" |
70 |
|
71 |
5. We could have the eclasses convert .egg-info into the newer .dist- |
72 |
info format. However, I'm not aware of any existing tool doing such |
73 |
a conversion, and I'm not convinced I want to write one right now, |
74 |
and whether it wouldn't have compatibility implications. |
75 |
|
76 |
These are all the options I can think of right now that don't make my |
77 |
head explode. I'd like to hear your ideas. |
78 |
|
79 |
-- |
80 |
Best regards, |
81 |
Michał Górny |