[gentoo-python] Re: distutils-r1: a bit of clean up + parallel builds

[Mike Gilbert]

On Thu, Nov 29, 2012 at 6:31 AM, Michał Górny <mgorny@g.o> wrote:
> 1) setup.py installs files to intermediate root (alike python.eclass).
>
> This way anything we do on the installed files doesn't collide with
> other merges potentially running in parallel. This also means that we
> don't have to delay installing the wrapper till all setup.py invocations
> have completed.
>
> This is done directly in distutils-r1_python_install. The setup.py is
> given a different --root, the renaming is done on intermediate image
> and the image is quickly merged to the destination.
>
> In order to perform the merge efficiently, I used:
>
>         cp --archive --link --no-clobber
>
> so that copy should preserve everything and use hard-links whenever
> possible. --no-clobber is necessary to avoid error on colliding files
> (cp doesn't want to overwrite when hardlinking).
>
>
> 2) the wrapper is installed in distutils-r1_python_install.
>
> Previously, distutils-r1_python_install only renamed the installed
> executables (because of distutils no-clobber behavior),
> and distutils-r1_python_install_all installed the wrapper.
>
> Now we can install both in the same function, since distutils installs
> into intermediate images. Therefore, the wrapper being installed
> in another intermediate image or even the real image won't collide.
>
>
> 3) the sub-phases are run in parallel.
>
> Since distutils itself is unable to do parallel builds, building Python
> packages with C extensions for multiple Python implementations can get
> very slow. In order to circumvent that, we're using the multiprocessing
> eclass to run sub-phases in parallel.
>
> This means that with 4 implementations enabled and -j4, all four
> implementations will be built at the same time. And if they have C
> extensions, 4 source files will be built at the same time. This also
> makes it possible to use distcc.
>
> As stated in the last patch:
>
> dev-python/lxml-3.0.1 for py2.6+2.7+3.2+3.3:
>
> - non-parallel: 11 min 23 sec
> - parallel: 7 min 49 sec (with a bit of swapping)
> - parallel w/ distcc: 3 min 40 sec
>
> main machine: Core2 2x1.6 GHz and almost 2 GiBs of RAM
> distcc host: Athlon64 2x2 GHz and 3 GiBs of RAM
>

I was just thinking to myself last night that parallel builds/installs
would be nice. You must be psychic.

My thought was to add a flag/variable for python_foreach_impl, but
your method should work just as well.

Building the temporary image in sub-directories of ${D} feels a bit
strange, but I guess it works. I think distutils.eclass used ${T}?

Anyway, +1 from me.