Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Michał Górny]

On Wed, 14 Dec 2016 15:27:25 +0300
Andrew Savchenko <bircoph@g.o> wrote:

> On Tue, 13 Dec 2016 10:36:15 +0100 Michał Górny wrote:
> > Introduce get_nproc(), a portable 'nproc' wrapper. It uses either
> > 'nproc' or a fallback Python multiprocessing module call to attempt to
> > determine the number of available processing units.
> >
> > This can be used e.g. to determine a safe number of jobs to run when
> > MAKEOPTS specifies unlimited --jobs and the build system in question
> > does not support --load-average.
> > ---
> >  eclass/multiprocessing.eclass | 25 +++++++++++++++++++++++++
> >  1 file changed, 25 insertions(+)
> >
> > diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
> > index 5a5fe9acb56a..0d241cdc15b6 100644
> > --- a/eclass/multiprocessing.eclass
> > +++ b/eclass/multiprocessing.eclass
> > @@ -53,6 +53,31 @@ bashpid() {
> >  	sh -c 'echo ${PPID}'
> >  }
> >
> > +# @FUNCTION: get_nproc
> > +# @USAGE: [${fallback:-1}]
> > +# @DESCRIPTION:
> > +# Attempt to figure out the number of processing units available.
> > +# If the value can not be determined, prints the provided fallback
> > +# instead. If no fallback is provided, defaults to 1.
> > +get_nproc() {
> > +	local nproc
> > +
> > +	if type -P nproc &>/dev/null; then
> > +		# GNU
> > +		nproc=$(nproc)
> > +	elif type -P python &>/dev/null; then
> > +		# fallback to python2.6+
> > +		# note: this may fail (raise NotImplementedError)
> > +		nproc=$(python -c 'import multiprocessing; print(multiprocessing.cpu_count());' 2>/dev/null)
>
> This is not portable. E.g. paludis users can have python-less
> system. Adding dev-lang/python to DEPEND will be also a bad idea,
> since this is quite heavy dependency.

You can bikeshed potential circumstances where it wouldn't work for
the next year. Which doesn't change that it would work quite reliably
for the most of Gentoo users.

> Since on Linux boxes nproc is from coreutils, which is in @system,
> so looks like only *bsd setups are the problem. On FreeBSD

...and all other operating systems (see: Prefix).

>   sysctl -a | egrep -i 'hw.ncpu'

I somehow doubt that would give me the expected number only, and I lack
a BSD install handy to test it.

> > +	fi
> > +
> > +	if [[ -n ${nproc} ]]; then
> > +		echo "${nproc}"
> > +	else
> > +		echo "${1:-1}"
> > +	fi
> > +}
> > +
> >  # @FUNCTION: makeopts_jobs
> >  # @USAGE: [${MAKEOPTS}]
> >  # @DESCRIPTION:

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>