[gentoo-cluster] examples of (large) Gentoo clusters

[John R. Dunning]

From: Bryan Green <bgreen@××××××××.gov>
    Date: Wed, 06 Dec 2006 16:33:12 -0800
    
    "John R. Dunning" writes:
    >     From: "Daniel van Ham Colchete" <daniel.colchete@×××××.com>
    >     Date: Tue, 5 Dec 2006 19:15:49 -0200
    >     
    >     Question: would you use Lustre 1.6 now or you would wait until the
    >     official version is out?
    > 
    > If I had to ship today, I'd probably ship the 1.6b5 code.  I find lustre 1.4
    > much more of a headache to configure and manage.  Thankfully, I don't have to
    > ship today; I expect by the time I do, cfs will have released the real 1.6
    > code.
    
    It is encouraging to hear that you are willing to base a product on Lustre
    1.6.  

There are problems either way, but based on my experience, I believe 1.6 is a
better choice, at least for the kind of situation I'm expecting to see.
That's based partly on the fact that in my testing I've seen a pretty small
quotient of out-and-out bugs (though there are a couple which are pretty
annoying) and partly on the fact that configuration and management-wise, 1.6
is way easier to deal with.  Part of what I expect will be happening in
deployments is to be building lustrefs's on the fly, under control of some
kind of configurator thingie.  For that kind of task, 1.4 would be much more
difficult to deal with.

We have a test gentoo cluster system which runs with lustre as its rootfs.  It
essentially "just works".  I've run numerous benchmarks and tests on it,
including bonnie, iozone, ltp, and assorted bits of application code; for the
most part it's been trouble-free, and the performance is generally pretty
good.  There are a few areas where, due to the properties of lustre, things
run unexpectedly slow, but for my purposes, they're all things that can be
lived with.  What I conclude from all that is that it's good enough for me to
consider shipping it as part of a product while still being able to sleep at
night :-}

	  Are you by any chance willing to share some of your knowledge about
    installing Lustre on Gentoo with others?  :)  

Sure.

Are you worrying about the kernel patching and other software installation
issues, or about how to set up the fs itself once you've got the software
together? 

Very briefly, the kernel-patching issue is an ongoing headache.  Lustre
patches vfs in non-trivial ways.  Unfortunately, everybody else does too.  It
becomes a fairly ugly patch-merging problem.  If you want, I can detail the
process I've settled on for coming up with a kernel patchset, but you won't
like it.  There are similar issues around ldiskfs and other bits, but they're
simpler, at least by comparison.  

Once the software is installed, configuring the fs goes pretty much by the
book.  mkfs.lustre, mount -t lustre, lfs, and lctl are your friends.  You'll
have some work to do deciding what your architecture is, in terms of how many
OSTs of what type, what's the interconnect topology which will get you the
best throughput etc, but there aren't really any landmines in there.  I've
only worked with the failover stuff a small amount, so can't really say a lot
about that, but the time I did play with it, it seemed to work as advertised.

If you are looking for more detail on something specific, I'm happy to say
what little I know about it.
						  
						  Perhaps I could make
    self-support an option, if it looked like it would be reliable.
    
Well, obviously, you should test the bejeezus out of your configuration before
you declare open season on it.  So far I haven't found reason to believe
lustre is substantially worse than any of the other open-source software
packages which are used in production situations.  I think that constitutes a
qualified "yes" :-}
-- 
gentoo-cluster@g.o mailing list