[gentoo-cluster] Re: Stats on HPC clusters running Gentoo

[Alexey Shvetsov <alexxy@gentoo.org>]

Hi all!

Sorry for delay.

Well we currently have few systems at SPbSTU (and yes some of them runs 
gentoo, but currently not that one that is in top500)

So about systems:

One old called Sigma

  8 nodes:
                 cpu:    4 x AMD Opteron 6174, 12 core, 2.2GHz
                 mem:    128G RAM
                 net:    40Gbps QDR Infiniband, 10Gbps Ethernet
                 Storage FS:     Lustre
                 Storage size:   13.5Tb

Second one is slightly exotic system called ccNUMA (actualy part of HPC 
complex installed in 2014)

64 nodes:
                 cpu:    3 x AMD Opteron 6380, 16 core, 2.5GHz
                 mem:    96G RAM
                 net:    56Gbps FDR Infiniband, NumaScale
                 Storage FS:     Lustre
                 Storage size:   1Pb

ccNUMA actuly uses NumaScale interconnect and bootloader so it can be 
used as Jumbo ccNUMA Single System Image
with ~3000 cores and 12Tb RAM that will be seen by single linux kernel 
(and yes it runs gentoo)


Other 3 partitions of our HPC systems currently runs CentOS 7 (but may 
be migrated to gentoo in near future). All this partitions shares same 
Lustre FS as ccNUMA

   * tornado     : nodes: 612
                     cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz
         cores/hwthreads: 28 / 28
                     mem: 64G
                     net: 56Gbps FDR Infiniband
   * tornado-k40 : nodes: 56
                     cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz
         cores/hwthreads: 28 / 28
            co-processor: 2 x Nvidia Tesla K40x
        co-porcessor mem: 12G
                     mem: 64G
                     net: 56Gbps FDR Infiniband
   * mic         : nodes: 288
                     cpu: 1 x Intel Xeon Phi 5120D @ 1.10GHz
         cores/hwthreads: 60 / 240
                     mem: 8G
                     net: 56Gbps FDR Infiniband

Storage: 1 PB Lustre FS



PS also we have some exotic storage system used for backup (400+ disks 
attached via SAS and runs single ZFS filesystem)

Andrew Savchenko писал 05-01-2018 02:18:
> Hi Alexxy and all!
> 
> While preparing Gentoo flyer[1,2] on upcoming FOSDEM event it was
> suggested to include some words about Gentoo in HPC. So I'm asking
> community about recent stats of using Gentoo on HPC clusters.
> 
> I ran several clusters at a university on Gentoo, but they are weak
> by modern standards (<10 TFlops CPU, no GPU for the best one, though
> with IB QDR interconnect), so I'm reluctant to add them in the
> flyer. If someone can provide us with better and actual numbers,
> this will be nice.
> 
> IIRC Alexxy is running some HPC setups. Are they on Gentoo? If yes,
> please provide some stats (Flops, number of cores, nodes, storage,
> interconnect and so on).
> 
> If someone can provide any other relevant info, we'll be grateful.
> 
> Years ago there was a Gentoo cluster project page with list of
> active setups, but I can't find it any more and it is likely to be
> outdated anyway.
> 
> [1]
> https://download.sumptuouscapital.com/gentoo/fosdem2018/gentoo-fosdem2018-flyer.pdf
> [2] git://git.sumptuouscapital.com/gentoo/pr-fosdem2018-flyer.git
> 
> Best regards,
> Andrew Savchenko

-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Best Regards,
Alexey 'Alexxy' Shvetsov, PhD
Department of Molecular and Radiation Biophysics
FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
Leningrad region, Gatchina, Russia
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru