Re: [gentoo-cluster] examples of (large) Gentoo clusters

[Bryan Green]

"John R. Dunning" writes:
>     From: Bryan Green <bgreen@××××××××.gov>
>
>     It is encouraging to hear that you are willing to base a product on Lustre
>     1.6.
>
> There are problems either way, but based on my experience, I believe 1.6 is a
> better choice, at least for the kind of situation I'm expecting to see.
> That's based partly on the fact that in my testing I've seen a pretty small
> quotient of out-and-out bugs (though there are a couple which are pretty
> annoying) and partly on the fact that configuration and management-wise, 1.6
> is way easier to deal with.  Part of what I expect will be happening in
> deployments is to be building lustrefs's on the fly, under control of some
> kind of configurator thingie.  For that kind of task, 1.4 would be much more
> difficult to deal with.
>

>From my limited experience with 1.6, and even more limited experience with 1.4, I
wholeheartedly agree with your assessment.  Version 1.4 looks like a real headache
to configure.  By comparison, 'mount -t lustre' pretty much characterizes the
simplicity of 1.6.

>
> 	  Are you by any chance willing to share some of your knowledge about
>     installing Lustre on Gentoo with others?  :)
>
> Sure.
>
> Are you worrying about the kernel patching and other software installation
> issues, or about how to set up the fs itself once you've got the software
> together?

Kernel patching.  For software installation, the lustre ebuild that was put on
this list recently seemed to do the trick for me, and setup was pretty easy.

I was able to patch the kernel, but the server was somewhat unstable.  Actually,
my memory is hazy.  I used the 'lustre-sources' ebuild, which effectively packaged
up the patches.  It was a 2.6.15 kernel.  I also tried to make a custom kernel for
lustre 1.4, but ultimately hit too many roadblocks.  I did learn a bit about how
to use 'quilt' though.

>
> Very briefly, the kernel-patching issue is an ongoing headache.  Lustre
> patches vfs in non-trivial ways.  Unfortunately, everybody else does too.  It
> becomes a fairly ugly patch-merging problem.  If you want, I can detail the
> process I've settled on for coming up with a kernel patchset, but you won't
> like it.  There are similar issues around ldiskfs and other bits, but they're
> simpler, at least by comparison.

I'd be interested in some of the details - off-list if that is more appropriate,
though it might be of interest to others on the list as well.  Once you download a
1.6 beta, how do you produce a kernel for Gentoo?  Do you patch a gentoo-sources
kernel, a vanilla-sources kernel, or something else?  The ideal would perhaps be
to have a 'lustre-sources' ebuild in the gentoo-science overlay.  :)

>
> 						  Perhaps I could make
>     self-support an option, if it looked like it would be reliable.
>
> Well, obviously, you should test the bejeezus out of your configuration before
> you declare open season on it.  So far I haven't found reason to believe
> lustre is substantially worse than any of the other open-source software
> packages which are used in production situations.  I think that constitutes a
> qualified "yes" :-}

Are you considering getting support from CFS at some point?  Sorry, you don't have
to answer if that is a sensitive question.  But part of this thread has been the
topic of encouraging CFS to support Gentoo.  Interestingly, my colleague, who is
in charge of installing Lustre (1.4) on our test system, is talking to CFS about
supporting a vanilla kernel configuration.  The reason?  We can't make the system
stable with a SLES kernel.  It was stable for a long time with Gentoo.  Now they
seem to have gotten it stable with SLES plus a vanilla 2.6.19 kernel (which of
course does not have the Lustre patches).  So they want Suse to provide a newer
SLES kernel with the Lustre patches, and CFS to support that configuration.

-bryan

-- 
gentoo-cluster@g.o mailing list