Re: [gentoo-user] Re: File system testing

[J. Roeleveld]

On Friday, September 19, 2014 01:41:26 PM James wrote:
> J. Roeleveld <joost <at> antarean.org> writes:
> > Out of curiosity, what do you want to simulate?
>
> subsurface flows in porous medium. AKA carbon sequestration
> by injection wells. You know, provide proof that those
> that remove hydrocarbons and actuall put the CO2 back
> and significantly mitigate the effects of their ventures.

Interesting topic. Can't provide advice on that topic.

> It's like this. I have been stuggling with my 17 year old "genius"
> son who is a year away from entering medical school, with
> learning responsibility. So I got him a hyperactive, highly
> intelligent (mix-doberman) puppy to nurture, raise, train, love
> and be resonsible for. It's one genious pup, teaching another
> pup about being responsible.

Overactive kids, always fun.
I try to keep mine busy without computers and TVs for now. (She's going to be 
3 in November)

> So goes the earl_bidness.......imho.
>
> > > Many folks are recommending to skip Hadoop/HDFS all  together
> >
> > I agree, Hadoop/HDFS is for data analysis. Like building a profile
> > about people based on the information companies like Facebook,
> > Google, NSA, Walmart, Governments, Banks,.... collect about their
> > customers/users/citizens/slaves/....
> >
> > > and go straight to mesos/spark. RDD (in-memory)  cluster
> > > calculations are at the heart of my needs. The opposite end of the
> > > spectrum, loads of small files and small apps; I dunno about, but, I'm
> > > all
> > > ears.
> > > In the end, my (3) node scientific cluster will morph and support
> > > the typical myriad  of networked applications, but I can take
> > > a few years to figure that out, or just copy what smart guys like
> > > you and joost do.....
> >
> >
> > Nope, I'm simply following what you do and provide suggestions where I
> > can.
> > Most of the clusters and distributed computing stuff I do is based on
> > adding machines to distribute the load. But the mechanisms for these are
> > implemented in the applications I work with, not what I design underneath.
> > The filesystems I am interested in are different to the ones you want.
>
> Maybe. I do not know what I want yet. My vision is very light weight
> workstations running lxqt (small memory footprint) or such, and a bad_arse
> cluster for the heavy lifting running on whatever heterogenous resoruces I
> have. From what I've read, the cluster and the file systems are all
> redundant that the cluster level (mesos/spark anyway) regardless of one any
> give processor/system is doing. All of Alans fantasies (needs) can be
> realized once the cluster stuff is master. (chronos, ansible etc etc).

Alan = your son? or?
I would, from the workstation point of view, keep the cluster as a single 
entity, to keep things easier.
A cluster FS for workstation/desktop use is generally not suitable for a High 
Performance Cluster (HPC) (or vice-versa)

> > I need to provided access to software installation files to a VM server
> > and access to documentation which is created by the users. The
> > VM server is physically next to what I already mentioned as server A.
> > Access to the VM from the remote site will be using remote desktop
> > connections.  But to allow faster and easier access to the
> > documentation, I need a server B at the remote site which functions as
> > described.  AFS might be suitable, but I need to be able to layer Samba
> > on top of that to allow a seamless operation.
> > I don't want the laptops to have their own cache and then having to
> > figure out how to solve the multiple different changes to documents
> > containing layouts. (MS Word and OpenDocument files).
>
> Ok so your customers (hperactive problem users) inteface to your cluster
> to do their work. When finished you write things out to other servers
> with all of the VM servers. Lots of really cool tools are emerging
> in the cluster space.

Actually, slightly different scenario.
Most work is done at customers systems. Occasionally we need to test software 
versions prior to implementing these at customers. For that, we use VMs.

The VM-server we have is currently sufficient for this. When it isn't, we'll 
need to add a 2nd VMserver.

On the NAS, we store:
- Documentation about customers + Howto documents on how to best install the 
software.
- Installation files downloaded from vendors (We also deal with older versions 
that are no longer available. We need to have our own collection to handle 
that)

As we are looking into also working from a different location, we need:
- Access to the VM-server (easy, using VPN and Remote Desktops)
- Access to the files (I prefer to have a local 'cache' at the remote location)

It's the access to files part where I need to have some sort of "distributed" 
filesystem.

> I think these folks have mesos + spark + samba + nfs all in one box. [1]
> [1]
> http://www.quantaqct.com/en/01_product/02_detail.php?mid=29&sid=162&id=163&q
> s=102

Had a quick look, these use MS Windows Storage 2012, this is only failover on 
the storage side. I don't see anything related to what we are working with.

> Build rather than purchase? WE have to figure out what you and Alan need, on
> a cluster, because it is what most folks need/want. It the admin_advantage
> part of cluster. (There also the Big Science (me) and Web centric needs.
> Right now they are realted project, but things will coalesce, imho. There
> is even "Spark_sql" for postgres admins [2].
>
>
> [2] https://spark.apache.org/sql/

Hmm.... that is interesting.

> > > > We use Lustre for our high performance general storage. I don't
> > > > have any numbers, but I'm pretty sure it is *really* fast (10Gbit/s
> > > > over IB sounds familiar, but don't quote me on that).
> > >
> > > AT Umich, you guys should test the FhGFS/btrfs combo. The folks
> > > at UCI swear about it, although they are only publishing a wee bit.
> > > (you know, water cooler gossip)...... Surely the Wolverines do not
> > > want those californians getting up on them?
> > >
> > > Are you guys planning a mesos/spark test?
> > >
> > > > > Personally, I would read up on these and see how they work. Then,
> > > > > based on that, decide if they are likely to assist in the specific
> > > > > situation you are interested in.
> > >
> > > It's a ton of reading. It's not apples-to-apple_cider type of reading.
> > > My head hurts.....
> >
> > Take a walk outside. Clear air should help you with the headaches :P
>
> Basketball, Boobs and Burbon use to work quite well. Now it's mostly
> basketball, but I'm working on someone "very cute"......

Cloning? Genetics?
Now that I am interested in. I could do with a couple of clones. ;)

Btw, there are women who know more about some aspects of IT then you and me 
put together. Some of those even manage to look great as well ;)

> > > I'm leaning to  DFS/LFS
> > > (2)  Luster/btrfs      and     FhGFS/btrfs
> >
> > I have insufficient knowledge to advise on either of these.
> > One question, why BTRFS instead of ZFS?
>
> I think btrfs has tremendous potential. I tried ZFS a few times,
> but the installs are not part of gentoo, so they got borked
> uEFI, grubs to uuids, etc etc also were in the mix. That was almost
> a year ago.

I did a quick test with Gentoo and ZFS. With the current documentation and 
ebuilds, it actually is quite simple to get to use. Provided you don't intend 
to use it for the root filesystem.

> For what ever reason the clustering folks I have
> read and communicated with are using ext4, xfs and btrfs. Prolly
> mostly because those are mostly used in their (systemd) inspired)
> distros....?

I think mostly because they are included native into the kernel and when 
dealing with HPC, you don't want to use a filesystem that is know to eat memory 
for breakfast.
When I switch the NAS over to ZFS, I will be using a dedicated machine with 
16GB of memory. Probably going to increase that to 32GB not too long after.

> > My current understanding is: - ZFS is production ready, but due to
> > licensing issues, not included in the kernel - BTRFS is included, but
> > not yet production ready with all planned features.
>
> Yep. the license issue with ZFS is a real killer for me. Besides,
> as an old state-machine, C hack, anything with B-tree is fabulous.
> Prejudices? Yep, but here, I'm sticking with my gut. Multi port
> ram can do mavelous things with Btree data structures. The
> rest will become available/stable. Simply, I just trust btrfs, in
> my gut.

I think both are stable and usable, with the limitations I currently see and 
confirmed by Rich.

> > For me, Raid6-like functionality is an absolute requirement and latest I
> > know is that that isn't implemented in BTRFS yet. Does anyone know when
> > that will be implemented and reliable? Eg. what time-frame are we
> > talking about?
>
> Now we are "communicating"! We have different visions. I want cheap,
> mirrored HD on small numbers of processors (less than 16 for now).
> I want max ram of the hightest performance possilbe. I want my reduncancy
> in my cluster with my cluster software deciding when/where/how-often
> to write out to HD. If the max_ram is not enought, then SSD will
> be between the ram and HD. Also, know this. The GPU will be assimilated
> into the processors, just like the FPUs were, some decade ago. Remember
> the i386 and the i387 math coprocessor chip? The good folks at opengl,
> gcc (GNU) and others will soon (eventually?) give us compilers to
> automagically use the gpu (and all of that blazingly fast ram therein,
> as slave to Alan admin authority (some bullship like that).

Yep, and for HPC and VMs, you want to keep as much memory available for what 
matters.
For a file storage cluster, memory is there to assist the serving of files. (As 
that is what matters there)

> So, my "Epiphany" is this. The bitches at systemd are to renamed
> "StripperD", as they will manage the boot cycle (how fast you can
> go down (save power) and come back up (online). The Cluster
> will rule off of your hardware, like a "Sheilk" "the ring that rules
> them all" be  the driver of the gabage collect processes.

Aargh, garbage collectors...
They tend to spring into action when least convenient...
Try to be able to control when they start cleaning.

> The cluster
> will be like the "knights of the round table"; each node helping, and
> standing for those other nodes (nobles) that stumble, always with
> extra resources, triple/quad redundancy and solving problems
> before that kernel based "piece of" has a chance to anything
> other than "go down" or "Come up" online.

Interesting, need to parse this slowly over the weekend.

> We shall see just who the master is of my hardawre!
> The sadest thing for me is that when I extolled about billion
> dollar companies corrupting the kernel development process, I did
> not even have those {hat wearing loosers} in mind. They are
> irrelevant. I was thinking about those semiconductor companies.
> You know the ones that accept billions of dollars for the NSA
> and private spoofs to embed hardware inside of hardware. The ones
> that can use "white noise" as a communications channel. The ones
> that can tap a fiber optic cable, with penetration. Those are
> the ones to focus on. Not a bunch of "silly boyz"......

For that, you need to keep the important sensitive data off the grid.

> My new K_main{} has highlighted a path to neuter systemd.
> But I do like how StripperD moves up and down, very quickly.

I don't care about boot times or shutdown times. If I did, I'd invest in high 
speed ram disks and SSDs.
Having 50 of the fastest SSDs in Raid-0 config will give more data then the 
rest of the system can handle ;)

If then using that for VMs which can keep the entire virtual disk also in 
memory, and you really are fllying with performance. That's why in-memory 
systems are becoming popular again.

> Cool huh?
> It's PARTY TIME!

Parties are nice... 

--
Joost