Re: [gentoo-dev] multiprocessing.eclass: doing parallel work in bash

[Mike Frysinger]

On Saturday 02 June 2012 00:11:19 Brian Harring wrote:
> On Fri, Jun 01, 2012 at 06:41:22PM -0400, Mike Frysinger wrote:
> > and put it into a new multiprocessing.eclass.  this way people can
> > generically utilize this in their own eclasses/ebuilds.
> >
> > it doesn't currently support nesting.  not sure if i should fix that.
> >
> > i'll follow up with an example of parallelizing of eautoreconf.  for
> > mail-filter/maildrop on my 4 core system, it cuts the time needed to run
> > from ~2.5 min to ~1 min.
>
> My main concern here is cleanup during uncontrolled shutdown; if the
> backgrounded job has hung itself for some reason, the job *will* just
> sit; I'm not aware of any of the PMs doing process tree killing, or
> cgroups containment; in my copious free time I'm planning on adding a
> 'cjobs' tool for others, and adding cgroups awareness into pkgcore;
> that said, none of 'em do this *now*, thus my concern.

i'm not sure there's much i can do here beyond adding traps

> > makeopts_jobs() {
>
> This function belongs in eutils, or somewhere similar- pretty sure
> we've got variants of this in multiple spots.  I'd prefer a single
> point to change if/when we add a way to pass parallelism down into the
> env via EAPI.

it's already in eutils.  but i'm moving it out of that and into this since it 
makes more sense in this eclass imo, and avoids this eclass from inheriting 
eutils.

> > multijob_child_init() {
> > 	[[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments"
> > 	trap 'echo ${BASHPID} $? >&'${mj_control_fd} EXIT
> > 	trap 'exit 1' INT TERM
> > }
>
> Kind of dislike this form since it means consuming code has to be
> aware of, and do the () & trick.
>
> A helper function, something like
> multijob_child_job() {
>   (
>   multijob_child_init
>   "$@"
>   ) &
>   multijob_post_fork || die "game over man, game over"
> }
>
> Doing so, would conver your eautoreconf from:
> for x in $(autotools_check_macro_val AC_CONFIG_SUBDIRS) ; do
>   if [[ -d ${x} ]] ; then
>     pushd "${x}" >/dev/null
>     (
>     multijob_child_init
>     AT_NOELIBTOOLIZE="yes" eautoreconf
>     ) &
>     multijob_post_fork || die
>     popd >/dev/null
>   fi
> done
>
> To:
> for x in $(autotools_check_macro_val AC_CONFIG_SUBDIRS) ; do
>   if [[ -d ${x} ]]; then
>     pushd "${x}" > /dev/null
>     AT_NOELIBTOOLIZE="yes" multijob_child_job eautoreconf
>     popd
>   fi
> done

it depends on the form of the code.  i can see both being useful.  should be 
easy to support both though:
multijob_child_init() {
	if [[ $# -eq 0 ]] ; then
		trap 'echo ${BASHPID} $? >&'${mj_control_fd} EXIT
		trap 'exit 1' INT TERM
	else
		(
		multijob_child_init
		"$@"
		) &
		multijob_post_fork || die
	fi
}

> Note, if we used an eval in multijob_child_job, the pushd/popd could
> be folded in.  Debatable.

i'd lean towards not.  keeps things simple and people don't have to get into 
quoting hell.

> > # @FUNCTION: multijob_finish_one
> > # @DESCRIPTION:
> > # Wait for a single process to exit and return its exit code.
> > multijob_finish_one() {
> >
> > 	[[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments"
> >
> > 	local pid ret
> > 	read -r -u ${mj_control_fd} pid ret
>
> Mildly concerned about the failure case here- specifically if the read
> fails (fd was closed, take your pick).

read || die ?  not sure what else could be done really.

> > multijob_finish() {
> > 	[[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments"
>
> Tend to think this should do cleanup, then die if someone invoked the
> api incorrectly; I'd rather see the children reaped before this blows
> up.

sounds good.  along those lines, i could add multijob_finish to 
EBUILD_DEATH_HOOKS so other `die` points also wait by default ...
-mike