[gentoo-user] Messed-up KMail filters

[gentoo-user] Messed-up KMail filters

[Peter Humphrey]

Greetings,

I noticed today that my mail filters weren't working right, so I had a look at 
them. I found 127 in place of the 30-odd there should be. Most of those are 
duplicates of good ones, and so I'm left with a dilemma: do I delete the lot 
and start from scratch; do I try to find an old backup with a good exported 
list of filters; or do I try to remove the rogues by hand? All those options 
are heavily error-prone.

By habit, I try to remember to export the filters to a text file whenever I make 
a change, so they should all be present in the text file if I haven't missed 
any.

Recovery from backup sounds easiest, and indeed I have weekly, monthly and 
yearly backups going back over some years. The complication is that the 
exported list of filters will be buried in a tar file of my /home partition, so 
I'll need to 'tar tf' each file and grep for KMailFilters.txt. Easiest perhaps, 
but I may lose the will to live while waiting. My own ~/ directory is 6GB 
today; half of /home.

Has anyone an idea? That nice Mr Google only casts gloom around. In my day, I 
might have written code to extract each filter into a file of its own, but that 
was 40-odd years ago. And the ToolbarName, the most easily recognised property 
of each filter, is somewhere in the middle of a block of 30-50 lines.

-- 
Regards,
Peter.

Re: [gentoo-user] Messed-up KMail filters

[Michael]

[-- Attachment #1: Type: text/plain, Size: 2072 bytes --]

On Monday, 22 September 2025 15:45:17 British Summer Time Peter Humphrey 
wrote:
> Greetings,
> 
> I noticed today that my mail filters weren't working right, so I had a look
> at them. I found 127 in place of the 30-odd there should be. Most of those
> are duplicates of good ones, and so I'm left with a dilemma: do I delete
> the lot and start from scratch; do I try to find an old backup with a good
> exported list of filters; or do I try to remove the rogues by hand? All
> those options are heavily error-prone.
> 
> By habit, I try to remember to export the filters to a text file whenever I
> make a change, so they should all be present in the text file if I haven't
> missed any.
> 
> Recovery from backup sounds easiest, and indeed I have weekly, monthly and
> yearly backups going back over some years. The complication is that the
> exported list of filters will be buried in a tar file of my /home partition,
> so I'll need to 'tar tf' each file and grep for KMailFilters.txt. Easiest
> perhaps, but I may lose the will to live while waiting. My own ~/ directory
> is 6GB today; half of /home.
> 
> Has anyone an idea? That nice Mr Google only casts gloom around. In my day,
> I might have written code to extract each filter into a file of its own,
> but that was 40-odd years ago. And the ToolbarName, the most easily
> recognised property of each filter, is somewhere in the middle of a block
> of 30-50 lines.

I'll leave the awk scripting for dedup to more capable contributors, but if 
this is an one-off problem it may be easier and quicker to run 'tar -tvf 
somearchive.tgz | grep  KMailFilters.txt' starting with the most recent backup 
archives.  At some point you'll come across an archive which has a smaller 
KMailFilters.txt file size than the current file.  Then you can extract and 
diff that file with your current version.

My suggestion falls apart if the duplication of filters was not caused by a 
single event, but has crept in progressively over the years.  In this case a 
deduplication script will be a better way to approach this.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Messed-up KMail filters

[Peter Humphrey]

On Monday, 22 September 2025 16:50:08 British Summer Time Michael wrote:

> I'll leave the awk scripting for dedup to more capable contributors, but if
> this is an one-off problem it may be easier and quicker to run 'tar -tvf
> somearchive.tgz | grep  KMailFilters.txt' starting with the most recent
> backup archives.  At some point you'll come across an archive which has a
> smaller KMailFilters.txt file size than the current file.  Then you can
> extract and diff that file with your current version.
> 
> My suggestion falls apart if the duplication of filters was not caused by a
> single event, but has crept in progressively over the years.  In this case a
> deduplication script will be a better way to approach this.

It seems I made a mountain out of a molehill. The problem did look pretty 
bleak, but the very first backup I tried had what I wanted: a good set of 38 
filters.

So thank you for your contribution, and apologies for requesting it. I should 
have tried before going into text.

-- 
Regards,
Peter.

Re: [gentoo-user] Messed-up KMail filters

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1573 bytes --]

On Tuesday, 23 September 2025 09:17:58 British Summer Time Peter Humphrey 
wrote:
> On Monday, 22 September 2025 16:50:08 British Summer Time Michael wrote:
> > I'll leave the awk scripting for dedup to more capable contributors, but
> > if
> > this is an one-off problem it may be easier and quicker to run 'tar -tvf
> > somearchive.tgz | grep  KMailFilters.txt' starting with the most recent
> > backup archives.  At some point you'll come across an archive which has a
> > smaller KMailFilters.txt file size than the current file.  Then you can
> > extract and diff that file with your current version.
> > 
> > My suggestion falls apart if the duplication of filters was not caused by
> > a
> > single event, but has crept in progressively over the years.  In this case
> > a deduplication script will be a better way to approach this.
> 
> It seems I made a mountain out of a molehill. The problem did look pretty
> bleak, but the very first backup I tried had what I wanted: a good set of 38
> filters.
> 
> So thank you for your contribution, and apologies for requesting it. I
> should have tried before going into text.

No need to apologise, a problem shared is a problem halved after all.  :-)

Also, your post could have enticed some useful script contribution applicable 
to other users' problems.  Now that you've solved this, it may be more 
interesting to ascertain what caused it in the first place.  Perhaps you used 
">>" instead of ">" in a terminal to restore your filters from a backup and 
thus appended to the original file instead of replacing it?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Messed-up KMail filters

[Peter Humphrey]

On Tuesday, 23 September 2025 09:31:19 British Summer Time Michael wrote:

> Now that you've solved this, it may be more interesting to ascertain what
> caused it in the first place. 

It is indeed a puzzle.

> Perhaps you used ">>" instead of ">" in a terminal to restore your filters
> from a backup and thus appended to the original file instead of replacing
> it?

Unlikely; I use KMail's built-in functions to save and restore filters. I only 
look at the saved file if I have a problem to solve, as in this case.

-- 
Regards,
Peter.

[SOLVED] Re: [gentoo-user] Messed-up KMail filters

[Peter Humphrey]

On Tuesday, 23 September 2025 14:38:43 Greenwich Mean Time Peter Humphrey 
wrote:
> On Tuesday, 23 September 2025 09:31:19 British Summer Time Michael wrote:
> > Now that you've solved this, it may be more interesting to ascertain what
> > caused it in the first place.
> 
> It is indeed a puzzle.
> 
> > Perhaps you used ">>" instead of ">" in a terminal to restore your filters
> > from a backup and thus appended to the original file instead of replacing
> > it?
> 
> Unlikely; I use KMail's built-in functions to save and restore filters. I
> only look at the saved file if I have a problem to solve, as in this case.

I think I know what happened. I've been using an external NVMe drive over 
USB-3, and it seems to have developed a fault of some kind, but progressively. 
I noticed recently that the case was cool, not the usual warm. I smelt a rat 
then, but it was only after many other little puzzles that the device finally 
died.

Replacement now in use.

-- 
Regards,
Peter.