1 |
On Mon, Jun 20, 2011 at 9:21 AM, Paul Hartman |
2 |
<paul.hartman+gentoo@×××××.com> wrote: |
3 |
> On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht <markknecht@×××××.com> wrote: |
4 |
>> Hi, |
5 |
>> Is split an appropriate program to use to break a single 10GB file |
6 |
>> into 100 100MB files to transfer over the net using rsync, and then |
7 |
>> use cat to reassemble? |
8 |
> |
9 |
> I think it should work just fine. I've split huge files into huge |
10 |
> chunks and never had any issues. |
11 |
> |
12 |
>> Is there some better way to do this? |
13 |
> |
14 |
> I wonder if splitting is even necessary; rsync will analyze the file |
15 |
> and only transmit the differences, right?. So I'd think that even if |
16 |
> the transfer fails, a retry would pick up where it left off (assuming |
17 |
> rsync keeps the failed copy). |
18 |
> |
19 |
> Also check out net-misc/unison. It seems to be designed for just this |
20 |
> sort of thing. |
21 |
|
22 |
I'll check them out Paul. Thanks for the extra ideas. |
23 |
|
24 |
I just tried it as an experiment between two machines here. Using |
25 |
split I broke the 10GB file into 100MB pieces, used rsync to get the |
26 |
pieces to my laptop, and then used cat to reassemble. The size of the |
27 |
results compares to the byte so that looks good. |
28 |
|
29 |
I was unsure whether rsync would restart where it left off or whether |
30 |
it would start over from the beginning. It's one huge file so it would |
31 |
be painful if it did the latter. This way I know my risk is at most |
32 |
100MB, or maybe 20MB if I break the original up into smaller pieces. |
33 |
|
34 |
Cheers, |
35 |
Mark |