Re: [gentoo-user] Copying data efficiently

[Mick]

On Saturday 17 May 2014 08:59:08 meino.cramer@×××.de wrote:
> Mick <michaelkintzios@×××××.com> [14-05-17 09:48]:
> > On Saturday 17 May 2014 04:33:57 meino.cramer@×××.de wrote:
> > > Hi,
> > >
> > > is there any tool in the Gentoo portage which may speed up (make it
> > > mopre efficient) the following task:
> > >
> > > On my HD there are data I want to copy to two identical external HDs.
> > > These HDs are of the same type/model and each is separately
> > > connectable via USB to my PC (...these two of the typical mobile
> > > external USB-HDs).
> > >
> > > Instead of copying the data twice from my PC to eah of the HDs I want
> > > to do it once...like I would be able to give the cp-command two
> > > instead of one target where to copy two.
> > >
> > > The result should be two identically populated external HDs, each of
> > > them useable/readable without the need to the other one.
> > >
> > > What tool of the portage tree I able to accomplish this task?
> > >
> > > Best regards,
> > > mcc
> >
> > 1. You could set up the two ext drives as a mirrored RAID and use your
> > copying/tar-ing/dd tool of choice.
> >
> >
> > 2. Or you could use pipe and tee to split the feed into any devices you
> > want,
> >
> > e.g.:
> >   pv /dev/sda1 | tee >(dd of=/dev/sdb1) >(dd of=/dev/sdc1)
> >
> > 3. Or you could use a sequential copy:
> >   cp -a /home /dev/sdb1/ && cp -a /home /dev/sdc1
> >
> > NOTES:
> >
> > a) Unlike other commands, pv will give you a progress bar so that you
> > know how long your back up is taking.
> >
> > b) The 3rd example above will copy sequentially, but depending on the
> > size of the file(s) the second copy may be read from cache.
> >
> > c) If you're doing this over the network then you can use nc in listening
> > mode at the receiving end, or ssh if the network is untrusted.
> >
> >
> > I'm interested to see what other ways will be suggested for this task.
>
> Hi Mick,
>
> thank your reply! :)
>
> From your numbering of the possibilities...
> 1.)  ...I am no RAID guru and would try this later with data, which
>     are not valuable...

If the drives are not permanently connected, then you will have to assemble 
the RAID each time you want to copy something, after you plug them in.  
Perhaps not the most efficient method.


> 2.) That looks interesting! Unfortunately it seems to copy device
>     contents on low level instead of files. The source are directory
>     structure -- not whole devices...

Well, I just offered an example for a whole drive.  You will need to point it 
to the file(s) you want to copy over.  The pv command works like cat; 

e.g. cat myfile | tee >(blah ...) 

is the same like:

pv myfile | tee >(blah ...)

>     Or did I overlook an option mentioned in the manpage...?
> 3.) The files I want to copy are in the size of some GB each. So the
>     cache isnt big enough to hold ALL files for the second part.
>
>
> Best regards,
> mcc

-- 
Regards,
Mick