| 1 |
Am Sonntag, den 28.12.2008, 14:12 +0000 schrieb Mick: |
| 2 |
> I am struggling to understand why this problem occurs: |
| 3 |
> |
| 4 |
> I scp a number of binary files (e.g. videos) from box A to box B. I have |
| 5 |
> bz2'd them beforehand. At box B I run bzip2 -tv file.bz2, which reports that |
| 6 |
> all is OK. |
| 7 |
> |
| 8 |
> Then I run bzip2 -dv file.bz2 but it tells me that some of the contents have |
| 9 |
> been corrupted. Fair enough, I run bzip2recover which runs its CRC checks |
| 10 |
> and does not report anything being wrong. So, I know run bzip2 -tv |
| 11 |
> rec00*file.bz2 and it finds a number of files with "data integrity (CRC) |
| 12 |
> error in data". |
| 13 |
> |
| 14 |
> I run bzip2recover on box A and scp afresh copies of the affected rec00* |
| 15 |
> files. Some of the rec00* files seem to be copied without further |
| 16 |
> corruption, but one or two report again corruption (on box B only). |
| 17 |
> |
| 18 |
> Unable to understand why this is so, I decide to use vanilla ftp to another |
| 19 |
> box (in a data center this time) to copy over the same files from box A. |
| 20 |
> Then run ftp get from box B. bzip2 -tv shows no more corruption (although |
| 21 |
> this is not universal - some files still show corruption when tested on box |
| 22 |
> B). |
| 23 |
> |
| 24 |
> My ISP applies traffic shaping (especially on the obscure ssh port that I have |
| 25 |
> chosen for the transmission) so some packets may be dropped, but this does |
| 26 |
> not explain why ftp occasionally works. |
| 27 |
|
| 28 |
> Can you explain why this seemingly random corruption occurs? |
| 29 |
Most likely a broken NIC or NIC-driver. I had this two times (once with |
| 30 |
a rented root server). Problems occurred only when connection speed was |
| 31 |
high. For further tests., try also using a slow connection (<1 mbit). |
| 32 |
bandwidth limiting is possible on the command line with scp and rsync. |
| 33 |
|
| 34 |
> Can I trust bzip2 and its CRC tests? |
| 35 |
I think you can, but _much_ better is using MD5 or SHAx checksums. Use: |
| 36 |
|
| 37 |
source:~> md5sum * > files.md5 |
| 38 |
(copy files and files.md5 over to dest.) |
| 39 |
destin:~> md5sum -c files.md5 |
| 40 |
|
| 41 |
Exchange md5sum for one of sha1sum, sha224sum, sha256sum, sha384sum, |
| 42 |
sha512sum. |
| 43 |
|
| 44 |
That's exactly what .deb, .rpm and gentoo does with their packages. |
| 45 |
|
| 46 |
Bye, |
| 47 |
Daniel |
Archives