Re: [gentoo-server] Re: [OT] Mirroring/backing-up a large

[dormando]

kashani wrote:
> A. Khattri wrote:
>> On Wed, 19 Apr 2006, kashani wrote:
>>
>>> I'm not sold on the Google approach.
>>>
>>>     Assuming someone was to build nine data servers we're talking
>>> roughly
>>> $3k per server (dual CPU, 4GB ram, raid 5 sata) or $30k with shipping
>>> and tax.
>>
>> Actually Google use the cheapest hardware they can find, buy in bulk, and
>> they assume stuff will fail so they plan accordingly. I very much doubt
>> they spend $3K per server...
>>
>
> Anyone trying to build the same with a purchase of less than 100 servers
> is not going to spend much less.
>
> 2 x 2GB RAM = $1000
> 2 x CPU = $500 or so
> Raid Card = $300
> Drives = $100 each
> 1u chassis/MB/etc = $500
>
> IIRC they drop the drives and shove everything into RAM. Which is fine
> when you have a limited data set and enough machines to shove it into
> RAM. Originally Google did have a limited data set. It was only after
> the infrastructure reached a critical size that they began Google mail
> and other large storage things. And had fours years to work out the
> operational kinks.
>
> I think it unlikely that someone with a single storage set has enough
> money or time to pay for a few Phds to write a custom filesystem, a few
> hundred servers (10TB/4GB), and the datacenter monkey necessary to
> replace gear constantly... oddly Google has both of these in spades. And
> if you read the whitepaper the smallest Google data cluster is nineteen
> servers, aka $40-60k for schleps like us, aka the cost of a SAN that
> that uses less power and burns less switch ports.
>
> This infatuation with the Google stuff that very few people (ie none of
> us) have the in house infrastructure to handle or the available cash is
> useless. Unless someone has actually built their own mini Google and
> wants to tell us all about it with nice numbers like total cost, source
> code, transactions per second, cost per GB of user data, throughput, and
> other data points.
>
> kashani

Sorry, I'll have to pipe up about that... There's at least one guy here 
using mogilefs, which is basically a google approach :) I'm using it as 
well.

There are a few nice things about distributing your files around:

- You don't necessarily need to dedicate machines to it. I have a 
hundred and change diskless boot webservers. I add two harddrives to a 
box, fire up the mogstored daemon, and put it back into the webserver pool.
- You're overspeccing. Why would each box need 4G of RAM? Whatever 
NAS/SAN you buy will *not* have nearly that much cache. If you're going 
to live without, live without :) A single dualcore chip or less could 
work too.
- Something like MogileFS uses node-level redundancy (NAID? Someone 
bothered giving it a name...), so there's no point in buying RAID cards. 
Use onboard SATA/PATA or the cheapest cards you can buy that will give 
decent throughput ($50 or less instead of $300).
- If it floats your boat, go ahead and get those 3U supermicro cases 
with 16 drive bays. Just use a couple cheap 4+ port SATA hot swap 
capable controllers instead of 2x$500+ battery backed RAID controllers. 
Get a bunch of them as slimmed down as possible. Hell, just fuse some 
steel together, tool an mb/PSU to it, and stack a ton of drives in front 
of some huge fans. Save 2/3rds of the case cost.

My case might not be enough like yours, but I can easily add terabytes 
of space for *just* the cost of the drives (and the relatively small 
power addage per box I add drives to). There are a few other services 
too though... Mogile needs a central DB (which you'd have two of, 
right?) and tracker services. Again, cheap is fine. Maybe you have a DB 
with some free resources... I just can't imagine someone selling me a 
NAS/SAN that's this scalable and this cheap.

have fun,
-Dormando
-- 
gentoo-server@g.o mailing list