Re: [gentoo-user] [OT] how to delete a directory tree really fast

[Rich Freeman]

On Fri, Oct 22, 2021 at 7:36 AM Helmut Jarausch <jarausch@××××××.be> wrote:
>
>
> There are more than 55,000 files on some <PREVBackUp> which is located
> on a BTRFS file system.
> Standard 'rm -rf' is really slow.
>
> Is there anything I can do about this?
>

I don't have any solid suggestions as I haven't used btrfs in a while.
File deletion speed is something that is very filesystem specific, but
on most it tends to be slow.

An obvious solution would be garbage collection, which is something
used by some filesystems but I'm not aware of any mainstream ones.
You can sort-of get that behavior by renaming a directory before
deleting it.  Suppose you have a directory created by a build system
and you want to do a new build.  Deleting the directory takes a long
time.  So, first you rename it to something else (or move it someplace
on the same filesystem which is fast), then you kick off your build
which no longer sees the old directory, and then you can delete the
old directory slowly at your leisure.  Of course, as with all garbage
collection, you need to have the spare space to hold the data while it
gets cleaned up.

I'm not sure if btffs is any faster at deleting snapshots/reflinks
than hard links.  I suspect it wouldn't be, but you could test that.
Instead of populating a directory with hard links, create a snapshot
of the directory tree, and then rsync over it/etc.  The result looks
the same but is COW copies.  Again, I'm not sure that btrfs will be
any faster at deleting reflinks than hard links though - they're both
similar metadata operations.  I see there is a patch in the works for
rsync that uses reflinks instead of hard links to do it all in one
command.  That has a lot of benefits, but again I'm not sure if it
will help with deletion.

You could also explore other filesystems that may or may not have
faster deletion, or look to see if there is any way to optimize it on
btrfs.

If you can spare the space, the option of moving the directory to make
it look like it was deleted will work on basically any filesystem.  If
you want to further automate it you could move it to a tmp directory
on the same filesystem and have tmpreaper do your garbage collection.
Consider using ionice to run it at a lower priority, but I'm not sure
how much impact that has on metadata operations like deletion.

-- 
Rich