1 |
On Sat, Sep 21, 2013 at 11:56 AM, Tanstaafl <tanstaafl@×××××××××××.org> wrote: |
2 |
> On 2013-09-20 6:43 PM, Paul Hartman <paul.hartman+gentoo@×××××.com> wrote: |
3 |
>> |
4 |
>> A couple weeks ago one of the drives died. I hot-swap replaced it with |
5 |
>> a new one (with no down-time) and the rebuild took exactly 10 hours. |
6 |
>> |
7 |
>> Under normal operation, the speed of the array for contiguous |
8 |
>> read/writes is about 600MB/sec, which is faster than my SSD (single |
9 |
>> drive, not RAIDed). |
10 |
> |
11 |
> |
12 |
> Thanks... |
13 |
> |
14 |
> But... RAID read/writes under normal operating conditions has nothing |
15 |
> whatsoever to do with REBUILD speeds/times. |
16 |
|
17 |
Of course, I just added that as additional info. |
18 |
|
19 |
Doing the numbers, my actual rebuild speed was roughly 83MB/sec average. |
20 |
|
21 |
> Again, the reason I'm interested in this is, if the rebuild times are |
22 |
> 'blindingly fast' (as compared to the times for SATA or even fast SAS drives |
23 |
> - ie, 1 hour vs your 10 hours)), then maybe a RAID6 with SSDs is back in the |
24 |
> realm of doable, since you don't lose 50% of available storage with RAID6... |
25 |
|
26 |
Mathematically, a 256GB drive will take 1/12th as long as a 3TB drive |
27 |
with all other factors being equal. Using the speed of my rebuild |
28 |
above, that would require less than an hour to rebuild a 256GB drive. |
29 |
I can only imagine SSDs or high-class HDDs would be even faster. So I |
30 |
think your goal of a 1-hour rebuild is a definite possibility, |
31 |
depending on your capacity needs and CPU/controller capabilities. |
32 |
|
33 |
You may need to tweak the RAID speed limit, cache settings, disable |
34 |
NCQ, enable read-ahead, etc. to realize the maximum speed depending on |
35 |
your particular hardware. There are dozens of pages online explaining |
36 |
how to speed up RAID syncs like that. Many people report seeing a 5x |
37 |
speed increase after making those adjustments, compared to linux |
38 |
default values. |
39 |
|
40 |
I searched for, but could not find, definitive references about SSD |
41 |
RAID build times. I found a lot of tweaker/overclocker type of sites |
42 |
bragging about their 3000MB/sec SSD RAID read speeds but no mention of |
43 |
replacing a failed drive. |
44 |
|
45 |
With SSDs, writes are considered the enemy, so using a pair of new |
46 |
SSDs in a mirrored raid is considered bad practice, because both |
47 |
drives will suffer the same number of writes, causing them both to |
48 |
reach their limit at around the same time. In that case it's |
49 |
considered safer to replace one of the working drives early, perhaps |
50 |
rotating a few extra working drives in and out every so often, to keep |
51 |
both sides of the mirror different ages or from varied manufacturing |
52 |
batches. |
53 |
|
54 |
Using SSDs in RAID5/6 also causes extra writes to occur for the |
55 |
parity, of course, but not as bad as mirroring, and you get the speed |
56 |
benefits from striping. One good thing with SSDs is that when they |
57 |
fail, it tends to fail on a write, so the chances of it failing to |
58 |
read when rebuilding in a RAID5/6 should be very small -- for HDDs |
59 |
that is the biggest fear during a rebuild. |
60 |
|
61 |
Enterprise SSDs can have 10x as many rated write/erase cycles than |
62 |
consumer SSDs (for nearly 10x the price), but even the cheapest SSD |
63 |
with 3000 write cycle lifetime should last you a hundred years if you |
64 |
write less than a few dozen GB a day to it. In a RAID5/6 you's |
65 |
spreading those writes out so it should last even longer, even with |
66 |
the parity overhead. |
67 |
|
68 |
If your RAID setup does not support passing the TRIM command to your |
69 |
SSDs it could cut your speed and lifetime down significantly. |
70 |
|
71 |
In the end it depends on your particular requirements and use case... |
72 |
as always. :) |
73 |
|
74 |
Here is a calculator that lets you plug in different drive sizes and |
75 |
rebuild speeds to see how long it will take, along with some other |
76 |
info: |
77 |
https://www.memset.com/tools/raid-calculator/ |
78 |
|
79 |
Good luck! |