Gentoo Logo
Gentoo Spaceship




Note: Due to technical difficulties, the Archives are currently not up to date. GMANE provides an alternative service for most mailing lists.
c.f. bug 424647
List Archive: gentoo-dev
Navigation:
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Headers:
To: gentoo-dev@g.o
From: Mike Frysinger <vapier@g.o>
Subject: Re: multiprocessing.eclass: doing parallel work in bash
Date: Sat, 2 Jun 2012 00:57:40 -0400
On Saturday 02 June 2012 00:11:19 Brian Harring wrote:
> On Fri, Jun 01, 2012 at 06:41:22PM -0400, Mike Frysinger wrote:
> > and put it into a new multiprocessing.eclass.  this way people can
> > generically utilize this in their own eclasses/ebuilds.
> > 
> > it doesn't currently support nesting.  not sure if i should fix that.
> > 
> > i'll follow up with an example of parallelizing of eautoreconf.  for
> > mail-filter/maildrop on my 4 core system, it cuts the time needed to run
> > from ~2.5 min to ~1 min.
> 
> My main concern here is cleanup during uncontrolled shutdown; if the
> backgrounded job has hung itself for some reason, the job *will* just
> sit; I'm not aware of any of the PMs doing process tree killing, or
> cgroups containment; in my copious free time I'm planning on adding a
> 'cjobs' tool for others, and adding cgroups awareness into pkgcore;
> that said, none of 'em do this *now*, thus my concern.

i'm not sure there's much i can do here beyond adding traps

> > makeopts_jobs() {
> 
> This function belongs in eutils, or somewhere similar- pretty sure
> we've got variants of this in multiple spots.  I'd prefer a single
> point to change if/when we add a way to pass parallelism down into the
> env via EAPI.

it's already in eutils.  but i'm moving it out of that and into this since it 
makes more sense in this eclass imo, and avoids this eclass from inheriting 
eutils.

> > multijob_child_init() {
> > 	[[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments"
> > 	trap 'echo ${BASHPID} $? >&'${mj_control_fd} EXIT
> > 	trap 'exit 1' INT TERM
> > }
> 
> Kind of dislike this form since it means consuming code has to be
> aware of, and do the () & trick.
> 
> A helper function, something like
> multijob_child_job() {
>   (
>   multijob_child_init
>   "$@"
>   ) &
>   multijob_post_fork || die "game over man, game over"
> }
> 
> Doing so, would conver your eautoreconf from:
> for x in $(autotools_check_macro_val AC_CONFIG_SUBDIRS) ; do
>   if [[ -d ${x} ]] ; then
>     pushd "${x}" >/dev/null
>     (
>     multijob_child_init
>     AT_NOELIBTOOLIZE="yes" eautoreconf
>     ) &
>     multijob_post_fork || die
>     popd >/dev/null
>   fi
> done
> 
> To:
> for x in $(autotools_check_macro_val AC_CONFIG_SUBDIRS) ; do
>   if [[ -d ${x} ]]; then
>     pushd "${x}" > /dev/null
>     AT_NOELIBTOOLIZE="yes" multijob_child_job eautoreconf
>     popd
>   fi
> done

it depends on the form of the code.  i can see both being useful.  should be 
easy to support both though:
multijob_child_init() {
	if [[ $# -eq 0 ]] ; then
		trap 'echo ${BASHPID} $? >&'${mj_control_fd} EXIT
		trap 'exit 1' INT TERM
	else
		(
		multijob_child_init
		"$@"
		) &
		multijob_post_fork || die
	fi
}

> Note, if we used an eval in multijob_child_job, the pushd/popd could
> be folded in.  Debatable.

i'd lean towards not.  keeps things simple and people don't have to get into 
quoting hell.

> > # @FUNCTION: multijob_finish_one
> > # @DESCRIPTION:
> > # Wait for a single process to exit and return its exit code.
> > multijob_finish_one() {
> > 
> > 	[[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments"
> > 	
> > 	local pid ret
> > 	read -r -u ${mj_control_fd} pid ret
> 
> Mildly concerned about the failure case here- specifically if the read
> fails (fd was closed, take your pick).

read || die ?  not sure what else could be done really.

> > multijob_finish() {
> > 	[[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments"
> 
> Tend to think this should do cleanup, then die if someone invoked the
> api incorrectly; I'd rather see the children reaped before this blows
> up.

sounds good.  along those lines, i could add multijob_finish to 
EBUILD_DEATH_HOOKS so other `die` points also wait by default ...
-mike
Attachment:
signature.asc (This is a digitally signed message part.)
Replies:
Re: multiprocessing.eclass: doing parallel work in bash
-- Cyprien Nicolas
References:
multiprocessing.eclass: doing parallel work in bash
-- Mike Frysinger
Re: multiprocessing.eclass: doing parallel work in bash
-- Brian Harring
Navigation:
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Previous by thread:
Re: multiprocessing.eclass: doing parallel work in bash
Next by thread:
Re: multiprocessing.eclass: doing parallel work in bash
Previous by date:
Re: Re: enhancement for doicon/newicon in eutils.eclass
Next by date:
Re: multiprocessing.eclass: doing parallel work in bash


Updated Jun 29, 2012

Summary: Archive of the gentoo-dev mailing list.

Donate to support our development efforts.

Copyright 2001-2013 Gentoo Foundation, Inc. Questions, Comments? Contact us.