Gentoo Logo
Gentoo Spaceship




Note: Due to technical difficulties, the Archives are currently not up to date. GMANE provides an alternative service for most mailing lists.
c.f. bug 424647
List Archive: gentoo-dev
Navigation:
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Headers:
To: Mike Frysinger <vapier@g.o>
From: Brian Harring <ferringb@...>
Subject: Re: multiprocessing.eclass: doing parallel work in bash
Date: Sat, 2 Jun 2012 16:59:02 -0700
On Fri, Jun 01, 2012 at 06:41:22PM -0400, Mike Frysinger wrote:
> # @FUNCTION: multijob_post_fork
> # @DESCRIPTION:
> # You must call this in the parent process after forking a child process.
> # If the parallel limit has been hit, it will wait for one to finish and
> # return the child's exit status.
> multijob_post_fork() {
> 	[[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments"
> 
> 	: $(( ++mj_num_jobs ))
> 	if [[ ${mj_num_jobs} -ge ${mj_max_jobs} ]] ; then
> 		multijob_finish_one
> 	fi
> 	return $?
> }

Minor note; the design of this (fork then check), means when a job 
finishes, we'll not be ready with more work.  This implicitly means 
that given a fast job identification step (main thread), and a slower 
job execution (what's backgrounded), we'll not breach #core of 
parallelism, nor will we achieve that level either (meaning 
potentially some idle cycles left on the floor).

Realistically, the main thread (what invokes post_fork) is *likely*, 
(if the consumer isn't fricking retarded) to be doing minor work- 
mostly just poking about figuring out what the next task/arguments 
are to submit to the pool.  That work isn't likely to be a full core 
worth of work, else as I said, the consumer is being a retard.

The original form of this was designed around the assumption that the 
main thread was light, and the backgrounded jobs weren't, thus it 
basically did the equivalent of make -j<cores>+1, allowing #cores 
background jobs running, while allowing the main thread to continue on 
and get the next job ready, once it had that ready, it would block 
waiting for a slot to open, then immediately submit the job once it 
had done a reclaim.

On the surface of it, it's a minor difference, but having the next 
job immediately ready to fire makes it easier to saturate cores.

Unfortunately, that also changes your API a bit; your call.

~harring


Replies:
Re: multiprocessing.eclass: doing parallel work in bash
-- Mike Frysinger
References:
multiprocessing.eclass: doing parallel work in bash
-- Mike Frysinger
Navigation:
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Previous by thread:
Re: multiprocessing.eclass: doing parallel work in bash
Next by thread:
Re: multiprocessing.eclass: doing parallel work in bash
Previous by date:
Re: multiprocessing.eclass: doing parallel work in bash
Next by date:
metadata/md5-cache


Updated Jun 29, 2012

Summary: Archive of the gentoo-dev mailing list.

Donate to support our development efforts.

Copyright 2001-2013 Gentoo Foundation, Inc. Questions, Comments? Contact us.