Re: [gentoo-cluster] Installing and using multiple MPI implementations at the same time.

[Alexander Piavka]

Hi Justin,

I've started playing with your empi implementation.

Some problems & suggestions:

1)'eselect mpi set ...' does not check for existance of ~/.env.d dir
and fails if one does not exists.

It creates ~/.env.d/mpi which looks like this:
----------------------
PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.1.2:/opt/blackdown-jdk-1.4.2.03/bin:/opt/blackdown-jdk-1.4.2.03/jre/bin"
MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man:/etc/java-config-2/current-system-vm/man:/usr/local/share/man:/usr/share/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.18/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.1.2/man:/opt/blackdown-jdk-1.4.2.03/man:/etc/java-config/system-vm/man/"
LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64:"
ESELECT_MPI_IMP="mpi-openmpi"
export LD_LIBRARY_PATH
export PATH
export MANPATH
export ESELECT_MPI_IMP
----------------------

while the following would be better:
----------------------
PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin:${PATH}"
MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man:${MANPATH}"
LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64:${LD_LIBRARY_PATH}"
ESELECT_MPI_IMP="mpi-openmpi"
export LD_LIBRARY_PATH
export PATH
export MANPATH
export ESELECT_MPI_IMP
----------------------

maybe even
----------------------
if [ "X${PATH}" != "X" ]; then
 	export PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin:${PATH}"
else
 	export PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin"
fi
if [ "X${MANPATH}" != "X" ]; then
 	export MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man:${MANPATH}"
else
 	export MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man"
fi
if [ "X${LD_LIBRARY_PATH}" != "X" ]; then
 	export LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64:${LD_LIBRARY_PATH}"
else
 	export LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64"
fi
export ESELECT_MPI_IMP
----------------------

  Also, probably . besides /etc/env.d/mpi/mpi-openmpi the /etc/env.d/XXmpi
file should also be created with the default empi profile then 'eselect mpi set <mpi-implementation>'
is run.

2)another problem is a failure to install binpkg 
of openmpi on another identical systems, the error is

  *
  * ERROR: mpi-openmpi/openmpi-1.2.5-r1 failed.
  * Call stack:
  *                 ebuild.sh, line 1717:  Called dyn_setup
  *                 ebuild.sh, line  768:  Called qa_call 'pkg_setup'
  *                 ebuild.sh, line   44:  Called pkg_setup
  *   openmpi-1.2.5-r1.ebuild, line   23:  Called mpi_pkg_setup
  *                mpi.eclass, line  306:  Called die
  * The specific snippet of code:
  *              [[ -f "${FILESDIR}"/${MPI_ESELECT_FILE} ]] \
  *                   || die "MPI_ESELECT_FILE is not defined/found. ${MPI_ESELECT_FILE}"
  *  The die message:
  *   MPI_ESELECT_FILE is not defined/found. eselect.mpi.openmpi
  *
  * If you need support, post the topmost build error, and the call stack if relevant.
  * A complete build log is located at '/var/tmp/portage/mpi-openmpi/openmpi-1.2.5-r1/temp/build.log'.
  *

I thinks this is due to MPI_ESELECT_FILE being defined in pkg_setup() of 
openmpi ebuild and not in top of ebuild (will check if this would help later)

3) If i have PORTDIR_OVERLAY="/usr/local/overlays/csbgu /tmp/empi"
empi --create --implementation mpi-openmpi =sys-cluster/openmpi-1.2.5-r1
  would create mpi-openmpi category tree under /usr/local/overlays/csbgu/mpi-openmpi
since it's first overlay in PORTDIR_OVERLAY, it would be nice if it could 
ALWAYS be created under the empi overlay i.e /tmp/empi/mpi-openmpi
Of couse i can put the empi overlay first in PORTDIR_OVERLAY instead
but i want to avoid manual tweaking as much as possible.
With all mpi-implementation residing in the same overlay tree as empi
it would much more convienient , for me, to auto distribute single overlay 
among cluster hosts and avoid possible need for commands
like 'empi --create --implementation mpi-openmpi ...'

  Thanks
  Alex

On Thu, 7 Feb 2008, Justin Bronder wrote:

> Well bug 44132 [1] has been a real thorn in my side for some time.  The
> problem being that all mpi implementations currently in portage block each
> other.  This is a real pain for any serious administrators, makes testing
> new/other implementations too time-consuming, and in my opinion has probably
> slowed any progress related to getting cluster applications into portage or
> the overlays.
>
> All that being said, I set out to do something about it.  Donnie Berkholz was
> nice enough to help me out early on and ended up suggesting that I look
> at crossdev.  I shamelessly ripped off the aforementioned script and tuned
> it for mpi implementations and packages.  It should be noted that all the
> changes I've implemented do [should, hopefully will] not effect the way the
> packages behave now, rather it just adds another option.  The idea is as
> follows.
>
> sys-cluster/empi:  Does the same stuff that crossdev does.  You create a new
> implementation root by specifying a name and mpi implementation package to
> build it with. [2]  Empi adds these to an overlay under a new category with
> the name you gave.  The ebuilds inherit mpi.eclass which handles pushing all
> the files to /usr/lib/mpi/<name>, and providing files for eselect-mpi.
>
> Later, you can add packages by specifying the category name and packages to
> install.  Empi grabs the ones that inherit mpi (including deps), adds them to
> the overlay and emerges.  The deps stuff hasn't been tested all that much.
>
> app-admin/eselect-mpi:  Used to setup a user's environment to call a named
> implementation install.  Basically does this by dropping a bash-sourceable
> file in ${HOME}/.env.d/mpi.  The not-so-fun part is that the user then has to
> remember to source that file.  Also used a fair amount by mpi.eclass.
>
> I threw some quick notes showing commands and output up in my devspace. [3]
> If anyone is at all interested, please try this out and let me know what you
> think.  Even if you don't try it out but have ideas/comments/rants, I'm all ears.
>
> Current overlay has the following ready to be tried out.
> eselect-mpi/eselect-mpi-0.0.1
> sys-cluster/empi-0.1
> sys-cluster/hpl-1.0-r3
> sys-cluster/lam-mpi-7.1.4-r1
> sys-cluster/mpi-examples-1.0
> sys-cluster/openmpi-1.2.5-r1
>
> A couple of final words, hpl and mpi-examples currently wouldn't work without
> empi, mainly because I'm lazy :)  Also I still haven't figured out a good way
> to handle do{man,doc,www} etcetera, ideas are welcome.  There's still a fair
> amount of work to be done, but I wanted to see if anyone had any feedback
> regarding the general concept first before pushing on.
>
> You can pull the overlay from rsync://cold-front.ath.cx/mpi (for now...)
>
> Thanks,
>
>
> [1] http://bugs.gentoo.org/show_bug.cgi?id=44132
> [2] Currently openmpi-1.2.5-r1 or lam-mpi-7.1.4-r1
> [3] http://dev.gentoo.org/~jsbronder/README.empi.txt
>
> --
> Justin Bronder
>

-- 
gentoo-cluster@l.g.o mailing list