[gentoo-user] Re: rpm or deb package installs

[James]

Bill Kenworthy <billk <at> iinet.net.au> writes:


> rpm is just a wrapper around a an archive with instructions on how to
> build and or install it.  I have more experience with rpm's but I
> believe debs are the same.  Just unwrap your .rpm/.deb file of choice
> and install it manually (the binaries/code are usually in a zip inside
> the rpm).  You should in most cases also be able to get a source rpm
> (which I suspect you are talking about anyway, but binaries do work deps
> permitting.

Yes the point is to be able to rapidly/easily be able to test and evaluate
many cluster/hpc codes, and other codes too, before I open them up
and look at them in detail, manually.


> you can install rpm and then install your package via rpm - seem to
> remember doing this with .debs somehow too.  deps are a problem but
> usually workable.

This is the argument for setting up an entire RH or debian workstation
to quickly evaluate the codes to find that less than %5 which are fast
and worthy of a deeper look. There are dozens of 'schedulers' for clusters
and then the concept of a 'framework' is even more loose or unconstrained
and the myriad of associated codes (modules) that one can use. It fact many
are pushing 'write your own framework'. It's hard to
categorize codes into a discernible 'stack' when it comes to HPC or cluster
offerings. Apache says one thing, a guy writing a book says another. It's
like the wild wild west with lots of folks (mostly large corps) shoot from
the hip and shooting off at the mouth. Many idiots and deceivers in this
space; and I intend to benchmark, profile and expose some of the myths.

So then I'll unpack the sources onto a gentoo system for deep examination
and low level (profile ) testing and evaluation and maybe creating an
ebuild if I like what I see. Distributing bloatware across a cluster
or on a HPC system, is a fools errand, imho, but that is exactly what
many are doing. It's very hard to figure out if they are 'flim_flam' artists
or sincerely belief their own musings.


> and why set up a workstation? - this sort of thing is tailor made for
> vm's.  Create a base for your experiments with essential packages,
> settings etc, snapshot it (golden master) and then throwaway->restore
> when finished with that iteration.

Ultimately, I think a VM or containers environment would be keen. I
liked docker, initially, but, like systemd, it is now bloating  so much that
it is becoming an entire operating system on it's own. Docker is too much
for me to worry about and learn. CoreOS is still attractive to me, but
it probably too much for me to divert into, at this time.  I'm really more
after HPC (high performance computing of scientific codes) than clustering
of a bizzilion processes (like log/web file analytics). That sort of stuff
is not hard to distribute, hadoop is sufficient for those routine, boring
codes and processes that are not time_critical. I'll do some of those
ordinary  tasks  but many cluster/hpc solutions already work good enough for
routine admin style processes, imho.


Big Science solutions using cluster or HPC are still very fragmented
if you look at some low level performance data. It takes a very specialized
focus to solve a given category of Big Science codes if you want to approach
any sort of reasonable robustness. It's like you need
a custom designed (HPCC) High Performance Computational Cluster with
a unique (at least highly customized and tweaked) implementation for each of
the many different possible configuration solutions. This part is evolving
on a daily basis, so apologies if it seems sketchy because it is sketchy at
best. I think where I'll end up is a myriad of HPCC, architecturally unique,
running along side of each other, mostly asleep until needed. It's even more
'fluid' when you consider the GPU resources, arm64 processors and the new
integrated CISC chips like the amd APU and the Intel+fpga chips.


> There are package managers besides gentoo/portage that can do a source
> build/install and track the files on gentoo - though portage will not
> know about it (rpm is one :)

Hmmmm. Where do I read more about this?


> and lastly, what do mean error prone? - to me a manual install is the
> ONLY way you can build a proper ebuild that catches most of the
> problems.  In the (admittedly few) ebuilds I have done an occasional
> human error is nothing compared to system problems for a difficult
> package.

Error prone with manual steps, due to the large amount of codes I
am evaluating. Once I get down to less than 5% then the manual
processes are fine, if not superior, as I'm also reviewing the codes
of interest.  It's also me, if I do not impose self_structure, I get
sloppy with trying to manually speed things up. Also, if you go
back to the days of .configure and look at many different make files,
there is little coding consistency among different codes; boost and other
lib codes are the best you can hope for on consistency, imho. ymmv.

OK? Keep in mind that I'm an EE, hardware guy that started with discrete
transistors, native assemble and state machines; and C was a luxury, so my
perspectives do not often 'jive' with the entire OO world, but, I am trying
to get along....... [1]

> BillK

James

[1] http://www.phy.duke.edu/~rgb/Beowulf/c++_interview/c++_interview.html