[gentoo-user] Clusters on Gentoo ?

[James]

Howdy one and all,

Many see a world where clusters abound even for the small business and
resource capable enthusist [1]. Clusters of old PCs are the norm, but a slew
of new extremely low powered 64bit embedded systems, running embedded linux,
with ample ram (ddr4 even) and up to (8) SATA-3 ports will undoubtly
be the targets of aquistion by hobbyist around the world. Other with more
salient goals are sure to follow!


For example, we (Gentoo) have just had one of the "titans" of the embedded
linux world, return to Gentoo. Linaro is the default industry group that
is leading the charge in new development for linux based embedded system
sharing most of their work with the larger open source communities.
Thomas Gall aka. tgall is working for Linaro as the acting director of the
Linaro Mobile Group [8,9].  Clusters will seemlessly integrate CPUs, GPUs,
Arms, FPGA, SOCs and many other instantiations of computational resources,
sooner rather than later. The Billion dollar players already run these sorts
of amalgamations for a very wide variety of reasons, so why should't the
bands of linux_commoners have access to such raw power? [10]


In a recent thread (schedulers) it was noted that several folks had interest
in clusters (privately operated clouds) as more than a passing interest.
Companion projects, such as Apache's "Spark" [4] have tremendous potential
as aggressive solutions such diverse fields as social media relationships,
distributed database techniques and new, massively parallel programing
paradymes for computationally intensive scientific endeavors, just to
mention a few [5,6,7].


So I'm soliciting the readers of this list to post any references to
distributed/cluster/cloud softwares/fileSystems they are aware of, have used
or would like to see; to guage interest in Mesos, Chronos, Spark (apache) as
well as all other open source cluster (distributed)  systems or tools [2].
My collection of such is sporadic, at best, and serves mostly my
math/science needs. Project Aethna, is one of the oldest efforts, still
kicking at MIT, the last I heard [3]. Newer/cooler efforts?


Hopefully, we can all share ideas and brainstorm about how Gentoo users
can lead the pack of linux distros into this brave_new world. [Overlays?]


curiously,
James



[1]
http://www.forbes.com/sites/marcochiappetta/2014/07/31/amd-opteron-64-bit-arm-based-seattle-dev-kits-are-shipping/?partner=yahootix

[2] http://hadoop.apache.org/docs/r1.2.1/cluster_setup.html

[3] https://ist.mit.edu/athena

[4] https://spark.apache.org/docs/latest/index.html

[5] https://spark.apache.org/docs/latest/graphx-programming-guide.html#overview

[6] http://en.wikipedia.org/wiki/Apache_Hadoop

[7] http://www.wired.com/2012/04/amazon-takes-genomics-research-to-the-clouds/

[8] http://www.gossamer-threads.com/lists/gentoo/dev/289556

[9] http://www.linaro.org/

[10] http://opencores.org/