1 |
1) setup.py installs files to intermediate root (alike python.eclass). |
2 |
|
3 |
This way anything we do on the installed files doesn't collide with |
4 |
other merges potentially running in parallel. This also means that we |
5 |
don't have to delay installing the wrapper till all setup.py invocations |
6 |
have completed. |
7 |
|
8 |
This is done directly in distutils-r1_python_install. The setup.py is |
9 |
given a different --root, the renaming is done on intermediate image |
10 |
and the image is quickly merged to the destination. |
11 |
|
12 |
In order to perform the merge efficiently, I used: |
13 |
|
14 |
cp --archive --link --no-clobber |
15 |
|
16 |
so that copy should preserve everything and use hard-links whenever |
17 |
possible. --no-clobber is necessary to avoid error on colliding files |
18 |
(cp doesn't want to overwrite when hardlinking). |
19 |
|
20 |
|
21 |
2) the wrapper is installed in distutils-r1_python_install. |
22 |
|
23 |
Previously, distutils-r1_python_install only renamed the installed |
24 |
executables (because of distutils no-clobber behavior), |
25 |
and distutils-r1_python_install_all installed the wrapper. |
26 |
|
27 |
Now we can install both in the same function, since distutils installs |
28 |
into intermediate images. Therefore, the wrapper being installed |
29 |
in another intermediate image or even the real image won't collide. |
30 |
|
31 |
|
32 |
3) the sub-phases are run in parallel. |
33 |
|
34 |
Since distutils itself is unable to do parallel builds, building Python |
35 |
packages with C extensions for multiple Python implementations can get |
36 |
very slow. In order to circumvent that, we're using the multiprocessing |
37 |
eclass to run sub-phases in parallel. |
38 |
|
39 |
This means that with 4 implementations enabled and -j4, all four |
40 |
implementations will be built at the same time. And if they have C |
41 |
extensions, 4 source files will be built at the same time. This also |
42 |
makes it possible to use distcc. |
43 |
|
44 |
As stated in the last patch: |
45 |
|
46 |
dev-python/lxml-3.0.1 for py2.6+2.7+3.2+3.3: |
47 |
|
48 |
- non-parallel: 11 min 23 sec |
49 |
- parallel: 7 min 49 sec (with a bit of swapping) |
50 |
- parallel w/ distcc: 3 min 40 sec |
51 |
|
52 |
main machine: Core2 2x1.6 GHz and almost 2 GiBs of RAM |
53 |
distcc host: Athlon64 2x2 GHz and 3 GiBs of RAM |