[gentoo-user] Re: Recommendations for scheduler

[James]

Joost Roeleveld <joost <at> antarean.org> writes:


> > Mesos looks promising for a variety of (Apache) reasons. Some key
> > technologies folks may want google about that are related:
> >
> > Quincy (fair schedular)
> > Chronos (scheduler)
> > Hadoop (scheduler)
>
> Hadoop not a scheduler. It's a framework for a Big Data clustered
> database.

> > HDFS (clusterd file system)
> Unless it's changed recently, not suitable for anything else then Hadoop
> and  contains a single point of failure.

I'm curious as to more information about this 'single point of failure. Can
you be more specific or provides links?

On this resource: 

http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html

JournalNode machines talks about surviving faults:

"increase the number of failures the system can tolerate, you should run an
odd number of JNs, (i.e. 3, 5, 7, etc.). Note that when running with N
JournalNodes, the system can tolerate at most (N - 1) / 2 failures and
continue to function normally. "

>
> > http://gpo.zugaina.org/sys-cluster/apache-hadoop-common
> >
> > Zookeeper (Fault tolerance)
> > SPARK ( optimized for interative jobs where a datase is resued in many
> > parallel operations (advanced math/science and many other apps.)
> > https://spark.apache.org/
> >
> > Dryad  Torque   Mpiche2 MPI
> > Globus tookit
> >
> > mesos_tech_report.pdf
> >
> > It looks as though Amazon, google, facebook and many others
> > large in the Cluster/Cloud arena are using Mesos......?
> >
> > So let's all post what we find, particularly in overlays.
>
> Unless you are dealing with Big Data projects, like Google, Facebook,
Amazon,  big banks,... you don't have much use for those projects.

Many scientific applications are using the cluster (cloud) or big data
approach to all sorts of problems. Furthermore, as GPU and the new
Arm systems with dozens and dozens of cpu cores inside one computer become
readily available, the cluster-cloud (big data) approach will become much
more pervasive in the next few years, imho.

http://blog.rescale.com/reservoir-simulation-moves-to-the-cloud/

There are thousands of small companies needing reservoir simulation, not to 
mention the millions of folks working on carbon sequestration.....
Anything to do with Biological or Chemical Science is using or moving
to the Cloud-Clustered world. For me, a Cluster is just a cloud internally
managee, rather than outsourcing it to others; ymmv.


> Mesos looks like a nice project, just like Hadoop and related are also
> nice. But for most people, they are as usefull as using Exalytics.

I'm not excited about an Oracle solution to anything. Many of the folks
I know consult on moving technologies away from Oracle's spear of influence,
not limited to mysql; ymmv. I know of one very large communications company
that went broke and had to merge because of those ridiculous Oracle fees.
Caveat Emptor; long live Postresql.  


> A scheduler should not have a large set of dependencies that you wouldn't
> use otherwise. That makes Chronos a non-option to me.

Those other technologies are often useful to folks who would be attracted to
something like chronos.

> Martin's project looks promising, but doesn't store the schedules
> internally. For repeating schedules, like what Alan was describing, you
> need to put those into scripts and start those from an existing cron.
> Of the 2, I think improving Martin's project is the most likely option
> for me as it doesn't have additional dependencies and seems to be
> easily implemented.
> Joost

Understood.
Like others, I'll be curious to follow what develops out of Martin's work.

For me Chronos, Mesos and the other aforementioned technologies look to be
more viable; particularly if one is preparing for a clustered world with
CPUs, GPUs, SoCs and Arm machines distributed about the ethernet  as
resources to be scheduled and utilized in a variety of schema. It's the
quest for one-infrastructure to solve many problems where scenarios compete. 

Big data is not the only reason for cloud-clusters. Theoretically,
(Clustered) systems can have a far greater resource utilization of networked
resources than traditional (distributed) approaches. I grant you that this
is a work in progress, but I personally know of dozens of mathematically
complex distributed systems that are  migrating to the clustered approach
rather than something custom or traditionally distributed.

Granted, Cloud <--> Clustered <--> Distributed are all overlaping approaches
to big problems. I do appreciate the candor of this thread.


James