From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id EB261138206 for ; Fri, 12 Jan 2018 23:33:23 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 25EF7E0843; Fri, 12 Jan 2018 23:33:23 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id DDF26E0835 for ; Fri, 12 Jan 2018 23:33:22 +0000 (UTC) Received: from localhost (unknown [91.246.97.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: bircoph) by smtp.gentoo.org (Postfix) with ESMTPSA id 848AB335C50; Fri, 12 Jan 2018 23:33:19 +0000 (UTC) Date: Sat, 13 Jan 2018 02:32:59 +0300 From: Andrew Savchenko To: gentoo-cluster@lists.gentoo.org Cc: Alexey Shvetsov , k_f@gentoo.org Subject: Re: [gentoo-cluster] Re: Stats on HPC clusters running Gentoo Message-Id: <20180113023259.1b9a0edb8ab9559ed34c766a@gentoo.org> In-Reply-To: References: <20180105021803.c1cf3df2a48ff77d44aede0e@gentoo.org> X-Mailer: Sylpheed 3.6.0 (GTK+ 2.24.30; i686-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-cluster@lists.gentoo.org Reply-to: gentoo-cluster@lists.gentoo.org Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA512"; boundary="Signature=_Sat__13_Jan_2018_02_33_00_+0300_FIvU8wG.Mv3XIPGR" X-Archives-Salt: 52f41f37-eab1-4676-9b1e-a34a02114177 X-Archives-Hash: 7e01cd744053b29f8e1a40d4f167ded0 --Signature=_Sat__13_Jan_2018_02_33_00_+0300_FIvU8wG.Mv3XIPGR Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, 11 Jan 2018 21:42:37 +0300 Alexey Shvetsov wrote: > Hi all! >=20 > Sorry for delay. >=20 > Well we currently have few systems at SPbSTU (and yes some of them runs=20 > gentoo, but currently not that one that is in top500) >=20 > So about systems: >=20 > One old called Sigma >=20 > 8 nodes: > cpu: 4 x AMD Opteron 6174, 12 core, 2.2GHz > mem: 128G RAM > net: 40Gbps QDR Infiniband, 10Gbps Ethernet > Storage FS: Lustre > Storage size: 13.5Tb >=20 > Second one is slightly exotic system called ccNUMA (actualy part of HPC=20 > complex installed in 2014) >=20 > 64 nodes: > cpu: 3 x AMD Opteron 6380, 16 core, 2.5GHz > mem: 96G RAM > net: 56Gbps FDR Infiniband, NumaScale > Storage FS: Lustre > Storage size: 1Pb >=20 > ccNUMA actuly uses NumaScale interconnect and bootloader so it can be=20 > used as Jumbo ccNUMA Single System Image > with ~3000 cores and 12Tb RAM that will be seen by single linux kernel=20 > (and yes it runs gentoo) >=20 >=20 > Other 3 partitions of our HPC systems currently runs CentOS 7 (but may=20 > be migrated to gentoo in near future). All this partitions shares same=20 > Lustre FS as ccNUMA >=20 > * tornado : nodes: 612 > cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz > cores/hwthreads: 28 / 28 > mem: 64G > net: 56Gbps FDR Infiniband > * tornado-k40 : nodes: 56 > cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz > cores/hwthreads: 28 / 28 > co-processor: 2 x Nvidia Tesla K40x > co-porcessor mem: 12G > mem: 64G > net: 56Gbps FDR Infiniband > * mic : nodes: 288 > cpu: 1 x Intel Xeon Phi 5120D @ 1.10GHz > cores/hwthreads: 60 / 240 > mem: 8G > net: 56Gbps FDR Infiniband >=20 > Storage: 1 PB Lustre FS >=20 >=20 >=20 > PS also we have some exotic storage system used for backup (400+ disks=20 > attached via SAS and runs single ZFS filesystem) Thanks, Alexey! This information will be very useful, we'll include parts of it in the Gentoo's FOSDEM flyer. Best regards, Andrew Savchenko --Signature=_Sat__13_Jan_2018_02_33_00_+0300_FIvU8wG.Mv3XIPGR Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEY+sE+qMMduKVLm7WVllTuVNydWwFAlpZRawACgkQVllTuVNy dWzunQ/+IE72J7gy8GsuSCasz9hSvhN+sms7vaEYa2103khKTM3724NTUEVAPMxC lCxL/Oin95WiaMK77zjqBXPYxl2FFjfNxOENVxBN42pW1SqYU07vs2LC0E8cJcqu oMWOAnRUoG2qBo8UbWQ9qCPIWNB8niEcWGumI6q8l4K9FDsrxpHLF6HHeH+ETfSW DZrZDQaDyvYXNhOaNnSgwpjTibphe5VkrrwFRigJfoaPwRRINOOruc9R4R8KFOv5 8fc+adcJbERXA/K+EinJPDuR8P1pNzOWIsQr+y6JRr9nl6f1eVm9D1xFa+XJJ1ub a0QHb7aNYURf3TvfUdIaXzH43izUF1S+MWiWBX2rQQ04AyPpdXKTLQnINhBqWTkE KeFQrrFhxzg6rcfPjOAs8g6Vq1qqs3VV6umg46vktFfyabdui5ED8hM5SM1FhLzC QHdlviOauQAebq/oMwCx/81xFH+Ka2ZVUdWP84cI/iUI4nzzNG5DPRafI01ivRCX gZtTqdK3zbuzcptBDqUG6pynhNkk9N50vjjHyPgyIGODIOP4iduNxp5XLtF29Gsw bKx+o3EtbdJud3ULbRki/eCZqO9OcxqwgqwnWuqRSctIDbU8d+qiRU0A+wRQCICX g5Ztlma30PCmEdKo3y4hFtUG/wdYTifsFgNkPB7/p6ZaAukOifw= =6TgJ -----END PGP SIGNATURE----- --Signature=_Sat__13_Jan_2018_02_33_00_+0300_FIvU8wG.Mv3XIPGR--