Gentoo Archives: gentoo-user

From: Indi <thebeelzebubtrigger@×××××.com>
To: gentoo-user@l.g.o
Subject: Re: [gentoo-user] Use split to break up a 10GB file binary?
Date: Tue, 21 Jun 2011 11:06:15
Message-Id: 20110621104147.GA21247@gaurahari.merseine.nu
In Reply to: Re: [gentoo-user] Use split to break up a 10GB file binary? by Mark Knecht
1 On Mon, Jun 20, 2011 at 05:28:05PM -0700, Mark Knecht wrote:
2 > On Mon, Jun 20, 2011 at 2:46 PM, Indi <thebeelzebubtrigger@×××××.com> wrote:
3 > > On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote:
4 > >> Hi,
5 > >>    Is split an appropriate program to use to break a single 10GB file
6 > >> into 100 100MB files to transfer over the net using rsync, and then
7 > >> use cat to reassemble?
8 > >>
9 > >>    Is there some better way to do this?
10 > >>
11 > >
12 > > Just using rsync by itself would probably be a great deal faster,
13 > > unless you have some undisclosed reason for wanting to split it up.
14 >
15 > Hi,
16 > Nothing technical that's undisclosed. My original reason was not
17 > knowing what rsync did in the case of errors I simply didn't want to
18 > start over on such a big file. I figured there was little to lose by
19 > stitching it back together are the other end and I could always figure
20 > out exactly what file had failed.
21 >
22 > That said I don't think there's much difference in the speed. In my
23 > case (and I think others will have a similar case) my uploads speeds
24 > are far lower than download. I get about 8MB/S download but only about
25 > 250KB/S upload. It's that low speed that's dominating everything else.
26 > When I first tried transferring the big file the intermediate speeds
27 > rsync was reporting were very similar.
28 >
29
30 Of course I was referring to the time taken by the extra steps, not the
31 transfer speed. :)
32
33 You might check man rsync though, it does what you need without
34 splitting and reassembling files..
35
36 --
37 caveat utilitor
38 ♫ ❤ ♫ ❤ ♫ ❤ ♫