[gentoo-project] Replacement hardware planning for Gentoo VM hosting

[Robin H. Johnson]

Infra has already discussed most of this hardware planning in
#gentoo-infra, but I thought it might be useful to see any other
comments on the hardware plan. If you wish to make private comments to
this thread, please send directly to infra@g.o or
gentoo-core@l.g.o instead of the gentoo-project list.

Remarks like 'you should use ZFS instead of this' aren't directly
helpful to this discussion. What is more useful is pointing out any
potential problems you might see with the plan, or gotchas in the
hardware.

We've previously run Ganeti [0] with general success, and we'd like to
continue doing so (vs libvirt or openstack). It offers VM storage
redundancy via DRBD (amongst other options), which we're going to take
best advantage of by using a cross-over 10Gbit link between two nodes
(as we have no 10GBit switching in the environment). Some of the VMs
will run on spinning disk, others on SSD, others maybe w/ dm-cache.
libvirt IS an easy fallback from Ganeti, but lacks some of the automated
failover and DRBD handling options.

This will house at least the following existing VMs, all of which have
large storage needs:
- woodpecker.gentoo.org
- roverlay.dev.g.o
- tinderbox.amd64.dev.g.o
- devbox.amd64.dev.g.o

And virtualize the following older systems:
[2007 Dells]
- finch.g.o (puppet)
- vulture.g.o (GSoC host)
[2010 Atoms]
- bellbird.g.o (infra services)
- bittern.g.o (blogs webhost)
- bobolink.g.o (rsync.g.o node, dns slave)
- brambling.g.o (bouncer, devmanual, infra-status)
[Other]
- meadowlark.g.o (infra services)

And New VMs/services:
- split git to rsync & snapshot generation from dipper?
- split blogs (and other) database hosting from dipper?

We'd probably keep the two other 2011 Dell systems in operation for the
moment, to distribute load better, but have enough capacity to run their
VMs as when they fail.

The general best prices we've seen are from a vendor that's new to us,
WiredZone, and we're willing to give them a try unless somebody has even
better pricing to offer us.

Hardware (all in $USD):
Supermicro SYS-2028TP-DECTR [1][2]
- $2,732.42/ea, quantity 1
- two half-width 2U nodes in a single chassis w/ shared redundant PSU.
- each node has:
- 2x 10GBe ports (there are no SFP options)
- 12x 2.5" SAS3, controller in JBOD/IT mode
Per node:
Intel Xeon E5-2620v4 [3] -
- $421.56/ea, quantity 2
32GB DDR4 PC4-19200 (2400MHz) 288-pin RDIMM ECC Registered [4],
- $162.89/ea, quantity 4
- require min of two DIMMs per CPU
- price jump to 64GB DIMMs very high.
- buy more RAM later?
Seagate 2TB SAS 12Gb/s 7200RPM 2.5in, ST2000NX0273 [5]
- $315.18/ea, quantity 4
- 4-disk RAID5 (mdadm)
Samsung 850 EVO 1TB, MZ-75E1T0B/AM [6]
- $345.00/ea, quantity 2
- RAID1 (mdadm)
= $3445.40/node

Overall cost:
$2,732.42 - chassis
$3,445.40 - left node components
$3,445.40 - right node components
$  315.18 - 1x spare ST2000NX0273 HDD
$   25.00 - 3ft CAT6a patch cable (estimated)

Parts sub-total: $9,963.40
Labour sub-total: $300 (estimate)
Taxes: $0.00 (Oregon has no sales taxes)
S&H: $200 (estimate)

Grant total: $10,463.40 (USD)

Future hardware improvement options:
- Add more RAM
- Add up to 6x more disks per node.

[0] http://www.ganeti.org/
[1] http://www.supermicro.com/products/system/2U/2028/SYS-2028TP-DECTR.cfm
[2] http://www.wiredzone.com/supermicro-multi-node-servers-twin-barebone-dual-cpu-2-node-sys-2028tp-dectr-10024389
[3] https://www.wiredzone.com/intel-components-cpu-processors-server-bx80660e52620v4-10025960
[4] https://www.wiredzone.com/supermicro-components-memory-ddr4-mem-dr432l-sl01-er24-10025993
[5] https://www.wiredzone.com/seagate-components-hard-drives-enterprise-st2000nx0273-10024175
[6] https://www.wiredzone.com/samsung-components-hard-drives-enterprise-mz-75e1t0b-am-10024043

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@g.o
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136