[gentoo-user] Re: File system testing

[James]

Alec Ten Harmsel <alec <at> alectenharmsel.com> writes:


> As far as HDFS goes, I would only set that up if you will use it for
> Hadoop or related tools. It's highly specific, and the performance is
> not good unless you're doing a massively parallel read (what it was
> designed for). I can elaborate why if anyone is actually interested.

Acutally, from my research and my goal (one really big scientific simulation
running constantly). Many folks are recommending to skip Hadoop/HDFS all
together and go straight to mesos/spark. RDD (in-memory)  cluster calculations
are at the heart of my needs. The opposite end of the spectrum, loads
of small files and small apps; I dunno about, but, I'm all ears.
In the end, my (3) node scientific cluster will morph and support
the typical myriad  of networked applications, but I can take
a few years to figure that out, or just copy what smart guys like
you and joost do.....


> We use Lustre for our high performance general storage. I don't have any
> numbers, but I'm pretty sure it is *really* fast (10Gbit/s over IB
> sounds familiar, but don't quote me on that).

AT Umich, you guys should test the FhGFS/btrfs combo. The folks 
at UCI swear about it, although they are only publishing a wee bit.
(you know, water cooler gossip)...... Surely the Wolverines do not
want those californians getting up on them?

Are you guys planning a mesos/spark test? 

> > Personally, I would read up on these and see how they work. Then,
> > based on that, decide if they are likely to assist in the specific
> > situation you are interested in.

It's a ton of reading. It's not apples-to-apple_cider type of reading.
My head hurts.....


I'm leaning to  DFS/LFS

(2)  Luster/btrfs      and     FhGFS/btrfs

Thoughts/comments?

James