Gentoo Archives: gentoo-user

From: Mark Knecht <markknecht@×××××.com>
To: gentoo-user@l.g.o
Subject: Re: [gentoo-user] Use split to break up a 10GB file binary?
Date: Mon, 20 Jun 2011 17:06:17
Message-Id: BANLkTi=jswbTVXy4xKfj8OJmh7vvAP+Bwg@mail.gmail.com
In Reply to: Re: [gentoo-user] Use split to break up a 10GB file binary? by Paul Hartman
1 On Mon, Jun 20, 2011 at 9:21 AM, Paul Hartman
2 <paul.hartman+gentoo@×××××.com> wrote:
3 > On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht <markknecht@×××××.com> wrote:
4 >> Hi,
5 >>   Is split an appropriate program to use to break a single 10GB file
6 >> into 100 100MB files to transfer over the net using rsync, and then
7 >> use cat to reassemble?
8 >
9 > I think it should work just fine. I've split huge files into huge
10 > chunks and never had any issues.
11 >
12 >>   Is there some better way to do this?
13 >
14 > I wonder if splitting is even necessary; rsync will analyze the file
15 > and only transmit the differences, right?. So I'd think that even if
16 > the transfer fails, a retry would pick up where it left off (assuming
17 > rsync keeps the failed copy).
18 >
19 > Also check out net-misc/unison. It seems to be designed for just this
20 > sort of thing.
21
22 I'll check them out Paul. Thanks for the extra ideas.
23
24 I just tried it as an experiment between two machines here. Using
25 split I broke the 10GB file into 100MB pieces, used rsync to get the
26 pieces to my laptop, and then used cat to reassemble. The size of the
27 results compares to the byte so that looks good.
28
29 I was unsure whether rsync would restart where it left off or whether
30 it would start over from the beginning. It's one huge file so it would
31 be painful if it did the latter. This way I know my risk is at most
32 100MB, or maybe 20MB if I break the original up into smaller pieces.
33
34 Cheers,
35 Mark

Replies

Subject Author
Re: [gentoo-user] Use split to break up a 10GB file binary? Michael Hampicke <gentoo-user@××××.biz>