| 1 |
On Thu, 11 Jan 2018 21:42:37 +0300 Alexey Shvetsov wrote: |
| 2 |
> Hi all! |
| 3 |
> |
| 4 |
> Sorry for delay. |
| 5 |
> |
| 6 |
> Well we currently have few systems at SPbSTU (and yes some of them runs |
| 7 |
> gentoo, but currently not that one that is in top500) |
| 8 |
> |
| 9 |
> So about systems: |
| 10 |
> |
| 11 |
> One old called Sigma |
| 12 |
> |
| 13 |
> 8 nodes: |
| 14 |
> cpu: 4 x AMD Opteron 6174, 12 core, 2.2GHz |
| 15 |
> mem: 128G RAM |
| 16 |
> net: 40Gbps QDR Infiniband, 10Gbps Ethernet |
| 17 |
> Storage FS: Lustre |
| 18 |
> Storage size: 13.5Tb |
| 19 |
> |
| 20 |
> Second one is slightly exotic system called ccNUMA (actualy part of HPC |
| 21 |
> complex installed in 2014) |
| 22 |
> |
| 23 |
> 64 nodes: |
| 24 |
> cpu: 3 x AMD Opteron 6380, 16 core, 2.5GHz |
| 25 |
> mem: 96G RAM |
| 26 |
> net: 56Gbps FDR Infiniband, NumaScale |
| 27 |
> Storage FS: Lustre |
| 28 |
> Storage size: 1Pb |
| 29 |
> |
| 30 |
> ccNUMA actuly uses NumaScale interconnect and bootloader so it can be |
| 31 |
> used as Jumbo ccNUMA Single System Image |
| 32 |
> with ~3000 cores and 12Tb RAM that will be seen by single linux kernel |
| 33 |
> (and yes it runs gentoo) |
| 34 |
> |
| 35 |
> |
| 36 |
> Other 3 partitions of our HPC systems currently runs CentOS 7 (but may |
| 37 |
> be migrated to gentoo in near future). All this partitions shares same |
| 38 |
> Lustre FS as ccNUMA |
| 39 |
> |
| 40 |
> * tornado : nodes: 612 |
| 41 |
> cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz |
| 42 |
> cores/hwthreads: 28 / 28 |
| 43 |
> mem: 64G |
| 44 |
> net: 56Gbps FDR Infiniband |
| 45 |
> * tornado-k40 : nodes: 56 |
| 46 |
> cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz |
| 47 |
> cores/hwthreads: 28 / 28 |
| 48 |
> co-processor: 2 x Nvidia Tesla K40x |
| 49 |
> co-porcessor mem: 12G |
| 50 |
> mem: 64G |
| 51 |
> net: 56Gbps FDR Infiniband |
| 52 |
> * mic : nodes: 288 |
| 53 |
> cpu: 1 x Intel Xeon Phi 5120D @ 1.10GHz |
| 54 |
> cores/hwthreads: 60 / 240 |
| 55 |
> mem: 8G |
| 56 |
> net: 56Gbps FDR Infiniband |
| 57 |
> |
| 58 |
> Storage: 1 PB Lustre FS |
| 59 |
> |
| 60 |
> |
| 61 |
> |
| 62 |
> PS also we have some exotic storage system used for backup (400+ disks |
| 63 |
> attached via SAS and runs single ZFS filesystem) |
| 64 |
|
| 65 |
Thanks, Alexey! |
| 66 |
|
| 67 |
This information will be very useful, we'll include parts of it in |
| 68 |
the Gentoo's FOSDEM flyer. |
| 69 |
|
| 70 |
Best regards, |
| 71 |
Andrew Savchenko |
Archives