Gentoo Archives: gentoo-python

From: "Michał Górny" <mgorny@g.o>
To: gentoo-python@l.g.o
Cc: python@g.o
Subject: [gentoo-python] distutils-r1: a bit of clean up + parallel builds
Date: Thu, 29 Nov 2012 11:30:24
Message-Id: 1354188694-7229-1-git-send-email-mgorny@gentoo.org
1 1) setup.py installs files to intermediate root (alike python.eclass).
2
3 This way anything we do on the installed files doesn't collide with
4 other merges potentially running in parallel. This also means that we
5 don't have to delay installing the wrapper till all setup.py invocations
6 have completed.
7
8 This is done directly in distutils-r1_python_install. The setup.py is
9 given a different --root, the renaming is done on intermediate image
10 and the image is quickly merged to the destination.
11
12 In order to perform the merge efficiently, I used:
13
14 cp --archive --link --no-clobber
15
16 so that copy should preserve everything and use hard-links whenever
17 possible. --no-clobber is necessary to avoid error on colliding files
18 (cp doesn't want to overwrite when hardlinking).
19
20
21 2) the wrapper is installed in distutils-r1_python_install.
22
23 Previously, distutils-r1_python_install only renamed the installed
24 executables (because of distutils no-clobber behavior),
25 and distutils-r1_python_install_all installed the wrapper.
26
27 Now we can install both in the same function, since distutils installs
28 into intermediate images. Therefore, the wrapper being installed
29 in another intermediate image or even the real image won't collide.
30
31
32 3) the sub-phases are run in parallel.
33
34 Since distutils itself is unable to do parallel builds, building Python
35 packages with C extensions for multiple Python implementations can get
36 very slow. In order to circumvent that, we're using the multiprocessing
37 eclass to run sub-phases in parallel.
38
39 This means that with 4 implementations enabled and -j4, all four
40 implementations will be built at the same time. And if they have C
41 extensions, 4 source files will be built at the same time. This also
42 makes it possible to use distcc.
43
44 As stated in the last patch:
45
46 dev-python/lxml-3.0.1 for py2.6+2.7+3.2+3.3:
47
48 - non-parallel: 11 min 23 sec
49 - parallel: 7 min 49 sec (with a bit of swapping)
50 - parallel w/ distcc: 3 min 40 sec
51
52 main machine: Core2 2x1.6 GHz and almost 2 GiBs of RAM
53 distcc host: Athlon64 2x2 GHz and 3 GiBs of RAM

Replies