Re: [gentoo-project] Replacement hardware planning for Gentoo VM hosting

[Matthew Thode]

On 05/19/2016 07:45 PM, Matthew Thode wrote:
> On 05/19/2016 07:42 PM, Matthew Thode wrote:
>> On 05/19/2016 06:47 PM, Robin H. Johnson wrote:
>>> Infra has already discussed most of this hardware planning in
>>> #gentoo-infra, but I thought it might be useful to see any other
>>> comments on the hardware plan. If you wish to make private comments to
>>> this thread, please send directly to infra@g.o or
>>> gentoo-core@l.g.o instead of the gentoo-project list.
>>>
>>> Remarks like 'you should use ZFS instead of this' aren't directly
>>> helpful to this discussion. What is more useful is pointing out any
>>> potential problems you might see with the plan, or gotchas in the
>>> hardware.
>>>
>>> We've previously run Ganeti [0] with general success, and we'd like to
>>> continue doing so (vs libvirt or openstack). It offers VM storage
>>> redundancy via DRBD (amongst other options), which we're going to take
>>> best advantage of by using a cross-over 10Gbit link between two nodes
>>> (as we have no 10GBit switching in the environment). Some of the VMs
>>> will run on spinning disk, others on SSD, others maybe w/ dm-cache.
>>> libvirt IS an easy fallback from Ganeti, but lacks some of the automated
>>> failover and DRBD handling options.
>>>
>>> This will house at least the following existing VMs, all of which have
>>> large storage needs:
>>> - woodpecker.gentoo.org
>>> - roverlay.dev.g.o
>>> - tinderbox.amd64.dev.g.o
>>> - devbox.amd64.dev.g.o
>>>
>>> And virtualize the following older systems:
>>> [2007 Dells]
>>> - finch.g.o (puppet)
>>> - vulture.g.o (GSoC host)
>>> [2010 Atoms]
>>> - bellbird.g.o (infra services)
>>> - bittern.g.o (blogs webhost)
>>> - bobolink.g.o (rsync.g.o node, dns slave)
>>> - brambling.g.o (bouncer, devmanual, infra-status)
>>> [Other]
>>> - meadowlark.g.o (infra services)
>>>
>>> And New VMs/services:
>>> - split git to rsync & snapshot generation from dipper?
>>> - split blogs (and other) database hosting from dipper?
>>>
>>> We'd probably keep the two other 2011 Dell systems in operation for the
>>> moment, to distribute load better, but have enough capacity to run their
>>> VMs as when they fail.
>>>
>>> The general best prices we've seen are from a vendor that's new to us,
>>> WiredZone, and we're willing to give them a try unless somebody has even
>>> better pricing to offer us.
>>>
>>> Hardware (all in $USD):
>>> Supermicro SYS-2028TP-DECTR [1][2]
>>> - $2,732.42/ea, quantity 1
>>> - two half-width 2U nodes in a single chassis w/ shared redundant PSU.
>>> - each node has:
>>> - 2x 10GBe ports (there are no SFP options)
>>> - 12x 2.5" SAS3, controller in JBOD/IT mode
>>> Per node:
>>> Intel Xeon E5-2620v4 [3] -
>>> - $421.56/ea, quantity 2
>>> 32GB DDR4 PC4-19200 (2400MHz) 288-pin RDIMM ECC Registered [4],
>>> - $162.89/ea, quantity 4
>>> - require min of two DIMMs per CPU
>>> - price jump to 64GB DIMMs very high.
>>> - buy more RAM later?
>>> Seagate 2TB SAS 12Gb/s 7200RPM 2.5in, ST2000NX0273 [5]
>>> - $315.18/ea, quantity 4
>>> - 4-disk RAID5 (mdadm)
>>> Samsung 850 EVO 1TB, MZ-75E1T0B/AM [6]
>>> - $345.00/ea, quantity 2
>>> - RAID1 (mdadm)
>>> = $3445.40/node
>>>
>>> Overall cost:
>>> $2,732.42 - chassis
>>> $3,445.40 - left node components
>>> $3,445.40 - right node components
>>> $  315.18 - 1x spare ST2000NX0273 HDD
>>> $   25.00 - 3ft CAT6a patch cable (estimated)
>>>
>>> Parts sub-total: $9,963.40
>>> Labour sub-total: $300 (estimate)
>>> Taxes: $0.00 (Oregon has no sales taxes)
>>> S&H: $200 (estimate)
>>>
>>> Grant total: $10,463.40 (USD)
>>>
>>> Future hardware improvement options:
>>> - Add more RAM
>>> - Add up to 6x more disks per node.
>>>
>>> [0] http://www.ganeti.org/
>>> [1] http://www.supermicro.com/products/system/2U/2028/SYS-2028TP-DECTR.cfm
>>> [2] http://www.wiredzone.com/supermicro-multi-node-servers-twin-barebone-dual-cpu-2-node-sys-2028tp-dectr-10024389
>>> [3] https://www.wiredzone.com/intel-components-cpu-processors-server-bx80660e52620v4-10025960
>>> [4] https://www.wiredzone.com/supermicro-components-memory-ddr4-mem-dr432l-sl01-er24-10025993
>>> [5] https://www.wiredzone.com/seagate-components-hard-drives-enterprise-st2000nx0273-10024175
>>> [6] https://www.wiredzone.com/samsung-components-hard-drives-enterprise-mz-75e1t0b-am-10024043
>>>
>>
>> +1 to this generally, the one question I have is if we want to spend ~1k
>> more on one of the 4x nodes, it'd allow us to expand easier in the
>> future.  The reasoning against this that I can think of is that we want
>> a higher disk/node ratio (which is just 6 per node instead of 12).
>>
> Another reason I just thought of is if we only have access to 15A
> outlets.  If so it'd limit us to one of the following (from our list) as
> they are 1600W.
>

http://www.supermicro.com/products/system/2U/2028/SYS-2028TR-HTR.cfm
http://www.supermicro.com/products/system/2U/2028/SYS-2028TR-H72R.cfm

fixed...

-- 
-- Matthew Thode (prometheanfire)