[gentoo-user] Re: Use split to break up a 10GB file binary?

[Francesco Talamona]

On Monday 20 June 2011, Paul Hartman wrote:
> On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht <markknecht@×××××.com>
wrote:
> > Hi,
> >   Is split an appropriate program to use to break a single 10GB
> > file into 100 100MB files to transfer over the net using rsync,
> > and then use cat to reassemble?
>
> I think it should work just fine. I've split huge files into huge
> chunks and never had any issues.
>
> >   Is there some better way to do this?
>
> I wonder if splitting is even necessary; rsync will analyze the file
> and only transmit the differences, right?. So I'd think that even if
> the transfer fails, a retry would pick up where it left off (assuming
> rsync keeps the failed copy).
>
> Also check out net-misc/unison. It seems to be designed for just this
> sort of thing.

Unison is wonderful for more complex tasks but is very inefficient with 
large files. As a matter of fact it uses rsync algorithm in order to get 
good performance, still isn't the best choice in this scenario, see: 
http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-
manual.html#speeding

I keep in sync the user home across two computer, and I use both of them 
daily. It would be impossible without unison, but the 20GB virtual 
machine is excluded from the sync.

Cheers
	Francesco

-- 
Linux Version 2.6.39-gentoo-r1, Compiled #1 SMP PREEMPT Thu Jun 9 
11:20:57 CEST 2011
Two 1GHz AMD Athlon 64 X2 Processors, 4GB RAM, 4021.84 Bogomips Total
aemaeth