From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id CACCB138330 for ; Thu, 11 Jan 2018 18:42:41 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 746D0E087C; Thu, 11 Jan 2018 18:42:40 +0000 (UTC) Received: from omrb.pnpi.spb.ru (omrb.pnpi.spb.ru [144.206.128.195]) by pigeon.gentoo.org (Postfix) with ESMTP id 083E7E085D for ; Thu, 11 Jan 2018 18:42:39 +0000 (UTC) Received: from omrb.pnpi.spb.ru (omrb.pnpi.spb.ru [144.206.128.195]) by omrb.pnpi.spb.ru (Postfix) with ESMTPSA id 245EDC19FD3; Thu, 11 Jan 2018 21:42:37 +0300 (MSK) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.99.2 at omrb Received: from [79.173.81.171] by omrb.pnpi.spb.ru with HTTP (HTTP/1.1 POST); Thu, 11 Jan 2018 18:42:37 +0000 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-cluster@lists.gentoo.org Reply-to: gentoo-cluster@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Thu, 11 Jan 2018 21:42:37 +0300 From: Alexey Shvetsov To: Andrew Savchenko Cc: gentoo-cluster@lists.gentoo.org, k_f@gentoo.org Subject: [gentoo-cluster] Re: Stats on HPC clusters running Gentoo In-Reply-To: <20180105021803.c1cf3df2a48ff77d44aede0e@gentoo.org> References: <20180105021803.c1cf3df2a48ff77d44aede0e@gentoo.org> Message-ID: X-Sender: alexxy@gentoo.org User-Agent: Roundcube Webmail/1.2.4 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,UNPARSEABLE_RELAY, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on omrb.pnpi.spb.ru X-Archives-Salt: 754f0ae5-a90c-4695-9f1b-d5461eb98cb3 X-Archives-Hash: c55ddeae02a93089368da521cd5a86fa Hi all! Sorry for delay. Well we currently have few systems at SPbSTU (and yes some of them runs gentoo, but currently not that one that is in top500) So about systems: One old called Sigma 8 nodes: cpu: 4 x AMD Opteron 6174, 12 core, 2.2GHz mem: 128G RAM net: 40Gbps QDR Infiniband, 10Gbps Ethernet Storage FS: Lustre Storage size: 13.5Tb Second one is slightly exotic system called ccNUMA (actualy part of HPC complex installed in 2014) 64 nodes: cpu: 3 x AMD Opteron 6380, 16 core, 2.5GHz mem: 96G RAM net: 56Gbps FDR Infiniband, NumaScale Storage FS: Lustre Storage size: 1Pb ccNUMA actuly uses NumaScale interconnect and bootloader so it can be used as Jumbo ccNUMA Single System Image with ~3000 cores and 12Tb RAM that will be seen by single linux kernel (and yes it runs gentoo) Other 3 partitions of our HPC systems currently runs CentOS 7 (but may be migrated to gentoo in near future). All this partitions shares same Lustre FS as ccNUMA * tornado : nodes: 612 cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz cores/hwthreads: 28 / 28 mem: 64G net: 56Gbps FDR Infiniband * tornado-k40 : nodes: 56 cpu: 2 x Intel Xeon CPU E5-2697 v3 @ 2.60GHz cores/hwthreads: 28 / 28 co-processor: 2 x Nvidia Tesla K40x co-porcessor mem: 12G mem: 64G net: 56Gbps FDR Infiniband * mic : nodes: 288 cpu: 1 x Intel Xeon Phi 5120D @ 1.10GHz cores/hwthreads: 60 / 240 mem: 8G net: 56Gbps FDR Infiniband Storage: 1 PB Lustre FS PS also we have some exotic storage system used for backup (400+ disks attached via SAS and runs single ZFS filesystem) Andrew Savchenko писал 05-01-2018 02:18: > Hi Alexxy and all! > > While preparing Gentoo flyer[1,2] on upcoming FOSDEM event it was > suggested to include some words about Gentoo in HPC. So I'm asking > community about recent stats of using Gentoo on HPC clusters. > > I ran several clusters at a university on Gentoo, but they are weak > by modern standards (<10 TFlops CPU, no GPU for the best one, though > with IB QDR interconnect), so I'm reluctant to add them in the > flyer. If someone can provide us with better and actual numbers, > this will be nice. > > IIRC Alexxy is running some HPC setups. Are they on Gentoo? If yes, > please provide some stats (Flops, number of cores, nodes, storage, > interconnect and so on). > > If someone can provide any other relevant info, we'll be grateful. > > Years ago there was a Gentoo cluster project page with list of > active setups, but I can't find it any more and it is likely to be > outdated anyway. > > [1] > https://download.sumptuouscapital.com/gentoo/fosdem2018/gentoo-fosdem2018-flyer.pdf > [2] git://git.sumptuouscapital.com/gentoo/pr-fosdem2018-flyer.git > > Best regards, > Andrew Savchenko -- Best Regards, Alexey 'Alexxy' Shvetsov Best Regards, Alexey 'Alexxy' Shvetsov, PhD Department of Molecular and Radiation Biophysics FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute, Leningrad region, Gatchina, Russia Gentoo Team Ru Gentoo Linux Dev mailto:alexxyum@gmail.com mailto:alexxy@gentoo.org mailto:alexxy@omrb.pnpi.spb.ru