Re: [gentoo-user] checksumming files

[Mick]

On Friday 05 December 2008, Albert Hopkins wrote:
> On Thu, 2008-12-04 at 07:10 +0000, Mick wrote:
> > Almost every time I split a large file >1G into say 200k chunks, then ftp
> > it to a server and then:
>
> That's thousands of files!  Have you gone mad?!

Ha! small error in units . . . it is 200M (of course this is no disclaimer of 
me going/gone mad . . .)  I think the server drops the connection above 230M 
file uploads or something like that, so I tried 200M files and it seems to 
work.

> >  cat 1 2 3 4 5 6 7 > completefile ; md5sum -c completefile
> >
> > if fails.  Checking the split files in turn I often find 1 or two chunks
> > that fail on their own md5 checks.  Despite that the concatenated file
> > often works (e.g. if it is a video file it'll play alright).
>
> Let me understand this. Are [1..7] the split files or the checksums of
> the split files?

They are the the split files which I concatenate into the complete file.

> If the former then 'md5sum -c completefile' will fail
> with "no properly formatted MD5 checksum lines found" or similar due to
> the fact that "completefile" is not a list of checksums.  If the latter,
> then how are you generating [1..7]? If you are using the split(1)
> command to split the files and are not passing at least "-a 3" to it
> then your file is going to be truncated do to the fact that the suffix
> length is too small to accommodate the thousands of files needed to
> split a 1GB+ file into 200k chunks. You should get an error like "split:
> Output file suffixes exhausted."
>
> Maybe if you give the exact commands used I might understand this
> better.
>
> I have a feeling that this is not the most efficient method of file
> transfer.

split --verbose -b 20000000 big_file

tnftp -r 45 -u 
ftp://<username>:<passwd>@<server_name>/htdocs/<directory_path>/ xaa xab xac 
xad . . .

The above would fail after xaa was uploaded and about 1/3 or less of xab.  So, 
I split up the individual file upload:

tnftp -r 45 -u 
ftp://<username>:<passwd>@<server_name>/htdocs/<directory_path>/ xaa ; sleep 
1m ; tnftp -r 45 -u 
ftp://<username>:<passwd>@<server_name>/htdocs/<directory_path>/ xab ; 
sleep ... ; etc.

Does this make sense?
-- 
Regards,
Mick