| 1 |
On Wednesday, September 17, 2014 04:20:24 PM Alec Ten Harmsel wrote: |
| 2 |
> As far as HDFS goes, I would only set that up if you will use it for |
| 3 |
> Hadoop or related tools. It's highly specific, and the performance is |
| 4 |
> not good unless you're doing a massively parallel read (what it was |
| 5 |
> designed for). I can elaborate why if anyone is actually interested. |
| 6 |
> |
| 7 |
> We use Lustre for our high performance general storage. I don't have |
| 8 |
any |
| 9 |
> numbers, but I'm pretty sure it is *really* fast (10Gbit/s over IB |
| 10 |
> sounds familiar, but don't quote me on that). |
| 11 |
|
| 12 |
I think any shared filesystem will be fast if you have a lot of bandwidth :) |
| 13 |
When comparing network filesystems it makes sense to keep the hardware |
| 14 |
identical reduce the overhead to a percentage. Eg. What is the theoretical |
| 15 |
maximum speed for the used network. (10Gbit/s) and what is the actual |
| 16 |
maximum speed you get with: |
| 17 |
1) a single really large file (200GB) |
| 18 |
2) a lot (100,000) smaller files (2MB) |
| 19 |
|
| 20 |
Then you can make an estimate on what to expect when using a 1Gbit/s |
| 21 |
network. I somehow don't expect James to have InfiniBand available for his |
| 22 |
research? |
| 23 |
Personally, when choosing between InfiniBand and Ethernet, I'm tempted |
| 24 |
to go with dedicated bonded 10Gbit/s links because of the price- |
| 25 |
difference. (A quick research shows me that Infiniband is about 3x as |
| 26 |
expensive for the same throughput) |
| 27 |
|
| 28 |
> > Personally, I would read up on these and see how they work. Then, |
| 29 |
> > based on that, decide if they are likely to assist in the specific |
| 30 |
> > situation you are interested in. |
| 31 |
> |
| 32 |
> Always good advice. |
| 33 |
|
| 34 |
It saves time to do some simple research (the reading type) before |
| 35 |
actually doing tests. |
| 36 |
|
| 37 |
-- |
| 38 |
Joost |
Archives