Gentoo Archives: gentoo-user

From: Alan McKinnon <alan.mckinnon@×××××.com>
To: gentoo-user@l.g.o
Subject: Re: [gentoo-user] OT: Fighting bit rot
Date: Tue, 08 Jan 2013 16:48:40
Message-Id: 20130108184234.65037a18@khamul.example.com
In Reply to: Re: [gentoo-user] OT: Fighting bit rot by Florian Philipp
1 On Tue, 08 Jan 2013 17:16:32 +0100
2 Florian Philipp <lists@×××××××××××.net> wrote:
3
4 > Am 08.01.2013 08:55, schrieb Alan McKinnon:
5 > > On Tue, 08 Jan 2013 08:27:51 +0100
6 > > Florian Philipp <lists@×××××××××××.net> wrote:
7 > >
8 > [...]
9 > >>
10 > >> As I said above, the point is that I need to detect the error as
11 > >> long as I still have a valid backup. Professional archive
12 > >> solutions do this on their own but I'm looking for something
13 > >> suitable for desktop usage.
14 > >
15 > > rsync might be able to give you something close to what you want
16 > > easily
17 > >
18 > > Use the -n switch for an rsync between your originals and the last
19 > > backup copy, and mail the output to yourself. Parse it looking for
20 > > ">" and "<" symbols and investigate why the file changed.
21 > >
22 > > This strikes me as being a very easy solution that you could use
23 > > reliably with a suitable combination of options.
24 > >
25 > >
26 >
27 > Hmm, good idea, albeit similar to the `md5sum -c`. Either tool leaves
28 > you with the problem of distinguishing between legitimate changes
29 > (i.e. a user wrote to the file) and decay.
30 >
31 > When you have completely static content, md5sum, rsync and friends are
32 > sufficient. But if you have content that changes from time to time,
33 > the number of false-positives would be too high. In this case, I
34 > think you could easily distinguish by comparing both file content and
35 > time stamps.
36 >
37 > Now, that of course introduces the problem that decay could occur in
38 > the same time frame as a legitimate change, thus masking the decay. To
39 > reduce this risk, you have to reduce the checking interval.
40
41
42 I think your basic problem is that you are trying to detect a rare
43 event (corruption) that looks exactly like a common event (edits you
44 intended to make)
45
46 I don't know how to tell these apart except by somehow recording which
47 files have been written to - inotify is useful for this - and removing
48 those from the list of things rsync says have changed.
49
50 All of which leads to a massively complex lump of code that is sure to
51 cause many more problems than it is designed to solve....
52
53 I'm afraid I don't have any real solution to offer.
54
55
56
57 --
58 Alan McKinnon
59 alan.mckinnon@×××××.com