[gentoo-user] Suboptimal emerging of python libraries with native code

[Ian Zimmerman]

I just emerged sci-libs/scipy.  I was unpleasantly surprised that the
job only loaded one of my cpus most of the time, resulting in a longish
wait (15-20 minutes).

I investigated a bit.  Originally, I had this in make.conf:

MAKEOPTS="-j5"

Then I had a look at the ebuild.  I saw that it inherits
multiprocessing.eclass and uses the makeopts_jobs function from it.
Here is the function:

makeopts_jobs() {
	[[ $# -eq 0 ]] && set -- "${MAKEOPTS}"
	# This assumes the first .* will be more greedy than the second .*
	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
	local jobs=$(echo " $* " | sed -r -n \
		-e 's:.*[[:space:]](-[a-z]*j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p' \
		-e "s:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:${2:-999}:p")
	echo ${jobs:-1}
}

Looking at this it seems to require at least one space between the -j
and the number!  Isn't that an unwarranted assumption?

So, I changed my make.conf to

MAKEOPTS="-j 5"

and rerun the build, but no joy - still no parallelization!  Another
closer look at the ebuild and I see this:

python_compile() {
	${EPYTHON} tools/cythonize.py || die
	distutils-r1_python_compile \
		$(usex python_targets_python3_5 "" "-j $(makeopts_jobs)") \
		${SCIPY_FCONFIG}
}

and sure enough, I'm building for python3.5, so I guess makeopts_jobs is
not even called and my MAKEOPTS are just ignored.  Why - what is so
special with python3.5 in the respect?  Would it help to switch to python3.6?

-- 
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.