On Thu, 11 Jan 2018 21:42:37 +0300 Alexey Shvetsov wrote:
> Hi all!
> 
> Sorry for delay.
> 
> Well we currently have few systems at SPbSTU (and yes some of them runs 
> gentoo, but currently not that one that is in top500)
> 
> So about systems:
> 
> One old called Sigma
> 
>   8 nodes:
>                  cpu:    4 x AMD Opteron 6174, 12 core, 2.2GHz
>                  mem:    128G RAM
>                  net:    40Gbps QDR Infiniband, 10Gbps Ethernet
>                  Storage FS:     Lustre
>                  Storage size:   13.5Tb
> 
> Second one is slightly exotic system called ccNUMA (actualy part of HPC 
> complex installed in 2014)
> 
> 64 nodes:
>                  cpu:    3 x AMD Opteron 6380, 16 core, 2.5GHz
>                  mem:    96G RAM
>                  net:    56Gbps FDR Infiniband, NumaScale
>                  Storage FS:     Lustre
>                  Storage size:   1Pb
> 
> ccNUMA actuly uses NumaScale interconnect and bootloader so it can be 
> used as Jumbo ccNUMA Single System Image
> with ~3000 cores and 12Tb RAM that will be seen by single linux kernel 
> (and yes it runs gentoo)
> 
> 
> Other 3 partitions of our HPC systems currently runs CentOS 7 (but may 
> be migrated to gentoo in near future). All this partitions shares same 
> Lustre FS as ccNUMA
> 
>    * tornado     : nodes: 612
>                      cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz
>          cores/hwthreads: 28 / 28
>                      mem: 64G
>                      net: 56Gbps FDR Infiniband
>    * tornado-k40 : nodes: 56
>                      cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz
>          cores/hwthreads: 28 / 28
>             co-processor: 2 x Nvidia Tesla K40x
>         co-porcessor mem: 12G
>                      mem: 64G
>                      net: 56Gbps FDR Infiniband
>    * mic         : nodes: 288
>                      cpu: 1 x Intel Xeon Phi 5120D @ 1.10GHz
>          cores/hwthreads: 60 / 240
>                      mem: 8G
>                      net: 56Gbps FDR Infiniband
> 
> Storage: 1 PB Lustre FS
> 
> 
> 
> PS also we have some exotic storage system used for backup (400+ disks 
> attached via SAS and runs single ZFS filesystem)

Thanks, Alexey!

This information will be very useful, we'll include parts of it in
the Gentoo's FOSDEM flyer.

Best regards,
Andrew Savchenko