1 |
On Friday 05 December 2008, Albert Hopkins wrote: |
2 |
> On Thu, 2008-12-04 at 07:10 +0000, Mick wrote: |
3 |
> > Almost every time I split a large file >1G into say 200k chunks, then ftp |
4 |
> > it to a server and then: |
5 |
> |
6 |
> That's thousands of files! Have you gone mad?! |
7 |
|
8 |
Ha! small error in units . . . it is 200M (of course this is no disclaimer of |
9 |
me going/gone mad . . .) I think the server drops the connection above 230M |
10 |
file uploads or something like that, so I tried 200M files and it seems to |
11 |
work. |
12 |
|
13 |
> > cat 1 2 3 4 5 6 7 > completefile ; md5sum -c completefile |
14 |
> > |
15 |
> > if fails. Checking the split files in turn I often find 1 or two chunks |
16 |
> > that fail on their own md5 checks. Despite that the concatenated file |
17 |
> > often works (e.g. if it is a video file it'll play alright). |
18 |
> |
19 |
> Let me understand this. Are [1..7] the split files or the checksums of |
20 |
> the split files? |
21 |
|
22 |
They are the the split files which I concatenate into the complete file. |
23 |
|
24 |
> If the former then 'md5sum -c completefile' will fail |
25 |
> with "no properly formatted MD5 checksum lines found" or similar due to |
26 |
> the fact that "completefile" is not a list of checksums. If the latter, |
27 |
> then how are you generating [1..7]? If you are using the split(1) |
28 |
> command to split the files and are not passing at least "-a 3" to it |
29 |
> then your file is going to be truncated do to the fact that the suffix |
30 |
> length is too small to accommodate the thousands of files needed to |
31 |
> split a 1GB+ file into 200k chunks. You should get an error like "split: |
32 |
> Output file suffixes exhausted." |
33 |
> |
34 |
> Maybe if you give the exact commands used I might understand this |
35 |
> better. |
36 |
> |
37 |
> I have a feeling that this is not the most efficient method of file |
38 |
> transfer. |
39 |
|
40 |
split --verbose -b 20000000 big_file |
41 |
|
42 |
tnftp -r 45 -u |
43 |
ftp://<username>:<passwd>@<server_name>/htdocs/<directory_path>/ xaa xab xac |
44 |
xad . . . |
45 |
|
46 |
The above would fail after xaa was uploaded and about 1/3 or less of xab. So, |
47 |
I split up the individual file upload: |
48 |
|
49 |
tnftp -r 45 -u |
50 |
ftp://<username>:<passwd>@<server_name>/htdocs/<directory_path>/ xaa ; sleep |
51 |
1m ; tnftp -r 45 -u |
52 |
ftp://<username>:<passwd>@<server_name>/htdocs/<directory_path>/ xab ; |
53 |
sleep ... ; etc. |
54 |
|
55 |
Does this make sense? |
56 |
-- |
57 |
Regards, |
58 |
Mick |