| 1 |
On Saturday 02 August 2014 16:53:26 James wrote: |
| 2 |
> Alan McKinnon <alan.mckinnon <at> gmail.com> writes: |
| 3 |
> > Well, we've found 2 projects that at least in part seek to achieve our |
| 4 |
> > general goals - chronos and Martin's new project. |
| 5 |
> > Why don't we both fool around with them for a bit and get a sense of |
| 6 |
> > what it will take to add features etc? Then we can meet back here and |
| 7 |
> > discuss. Always better to build on an existing foundation |
| 8 |
> |
| 9 |
> Mesos looks promising for a variety of (Apache) reasons. Some key |
| 10 |
> technologies folks may want google about that are related: |
| 11 |
> |
| 12 |
> Quincy (fair schedular) |
| 13 |
> Chronos (scheduler) |
| 14 |
> Hadoop (scheduler) |
| 15 |
|
| 16 |
Hadoop not a scheduler. It's a framework for a Big Data clustered database. |
| 17 |
|
| 18 |
> HDFS (clusterd file system) |
| 19 |
|
| 20 |
Unless it's changed recently, not suitable for anything else then Hadoop and |
| 21 |
contains a single point of failure. |
| 22 |
|
| 23 |
> http://gpo.zugaina.org/sys-cluster/apache-hadoop-common |
| 24 |
> |
| 25 |
> Zookeeper (Fault tolerance) |
| 26 |
> SPARK ( optimized for interative jobs where a datase is resued in many |
| 27 |
> parallel operations (advanced math/science and many other apps.) |
| 28 |
> https://spark.apache.org/ |
| 29 |
> |
| 30 |
> Dryad Torque Mpiche2 MPI |
| 31 |
> Globus tookit |
| 32 |
> |
| 33 |
> mesos_tech_report.pdf |
| 34 |
> |
| 35 |
> It looks as though Amazon, google, facebook and many others |
| 36 |
> large in the Cluster/Cloud arena are using Mesos......? |
| 37 |
> |
| 38 |
> So let's all post what we find, particularly in overlays. |
| 39 |
|
| 40 |
Unless you are dealing with Big Data projects, like Google, Facebook, Amazon, |
| 41 |
big banks,... you don't have much use for those projects. |
| 42 |
|
| 43 |
Mesos looks like a nice project, just like Hadoop and related are also nice. |
| 44 |
But for most people, they are as usefull as using Exalytics. |
| 45 |
|
| 46 |
A scheduler should not have a large set of dependencies that you wouldn't use |
| 47 |
otherwise. That makes Chronos a non-option to me. |
| 48 |
|
| 49 |
Martin's project looks promising, but doesn't store the schedules internally. |
| 50 |
For repeating schedules, like what Alan was describing, you need to put those |
| 51 |
into scripts and start those from an existing cron. |
| 52 |
|
| 53 |
Of the 2, I think improving Martin's project is the most likely option for me |
| 54 |
as it doesn't have additional dependencies and seems to be easily implemented. |
| 55 |
|
| 56 |
-- |
| 57 |
Joost |
Archives