Gentoo Archives: gentoo-dev

From: "Michał Górny" <mgorny@g.o>
To: gentoo-dev@l.g.o
Subject: [gentoo-dev] Looking for a solution to the distutils/setuptools .egg-info mess
Date: Mon, 10 Jan 2022 05:39:29
Message-Id: 908ab66e71703930054cdbfd1cf63eb888ea35e2.camel@gentoo.org
1 Hi, everyone.
2
3 TL;DR: how to deal with setuptools (and newer distutils vendored by
4 setuptools) replacing .egg-info files with directories?
5
6
7 I know I'm reiterating the same topic but I think we're coming to having
8 to actually do something about this and I still haven't found
9 a satisfactory solution.
10
11 For the people new to the problem, a quick recap: Python packages
12 install metadata called .egg-info (newer build systems use .dist-info
13 but that's another matter). Original Python distutils installed this
14 as a single file but setuptools replaced it with a directory.
15
16 Now that distutils is deprecated, setuptools is vendoring its own
17 version. We can switch between the system and the vendored version
18 using an envvar. By default, setuptools < 60 uses system distutils,
19 and >= 60 (masked) uses vendored distutils.
20
21 The big problem is that switching implies changing the format, so if you
22 install foo-X, then switch, then reinstall the same version of foo,
23 you're going to have the file replaced by a directory. This is not
24 supported by the PMS, and Portage handles it somewhat suboptimally
25 (renaming the old file and leaving it orphaned).
26
27 I should probably emphasize here that the .egg-info path contains
28 the package version, so this is a problem only if the same upstream
29 version is being reinstalled.
30
31 You can easily reproduce the problem by playing with:
32
33 SETUPTOOLS_USE_DISTUTILS=stdlib
34 SETUPTOOLS_USE_DISTUTILS=local # vendored
35
36 and repeatedly building some DISTUTILS_USE_SETUPTOOLS=no packages.
37
38
39 From a quick grep, there are 179 packages using DUS=no right now
40 (and there might be more using distutils without the declaration).
41 I don't think we can ignore the problem.
42
43 What we can do right now is force the SETUPTOOLS_USE_DISTUTILS=stdlib
44 default back via the eclass or patches for the time being. This will
45 let us finally unmask setuptools-60+ while delaying fixing the problem.
46 However, given pypa's tendency to remove deprecated stuff quickly, this
47 is unlikely to work for long.
48
49 Some ideas on fixing this:
50
51 1. For a start, with distutils deprecated are actually migrating to
52 setuptools or other build systems upstream. This effectively solves
53 the problem for us since the .egg-info switch happens on version bump
54 and there is no file collision. However, this isn't going to help for
55 dead projects.
56
57 2. We could control the distutils version in ebuilds directly,
58 i.e. force "stdlib" for the current versions and have developers switch
59 to "local" on version bumps. Combined with 1., this will probably
60 increase the coverage a bit but dead packages will remain in the way.
61 It also relies on all devs understanding the problem.
62
63 3. The developers could explicitly bump versions (i.e. create "Gentoo
64 subversions") of packages that don't expect any updates. We can get
65 100% coverage this way but it's hard and requires patching.
66
67 4. We could have the eclasses switch to "local" model and rename
68 the .egg-info files somehow at some point. The main question is "rename
69 how?"
70
71 5. We could have the eclasses convert .egg-info into the newer .dist-
72 info format. However, I'm not aware of any existing tool doing such
73 a conversion, and I'm not convinced I want to write one right now,
74 and whether it wouldn't have compatibility implications.
75
76 These are all the options I can think of right now that don't make my
77 head explode. I'd like to hear your ideas.
78
79 --
80 Best regards,
81 Michał Górny

Replies