Re: [gentoo-user] [OT] how to delete a directory tree really fast

[Miles Malone]

And honestly, expanding on what Rich said...  Given your particular
circumstances with the extensive number of hardlinks are pretty
specific, I reckon you might be best off just setting up a small scale
test of some options and profiling it.  Converting it all to a btrfs
subvolume might be a realistic option, or might take an order of
magnitude more time than just waiting for it all to delete.  or all of
the various move tricks mentioned previously

If this were an "I know I need to do this in the future, what should I
do" question then you'd either put it all in a subvolume to begin
with, or select the file system specifically for its speed deleting
small files... (Certainly dont quote me here, but wasnt JFS the king
of that back in the day?  I cant quite recall)

On Fri, 22 Oct 2021 at 22:29, Rich Freeman <rich0@g.o> wrote:
>
> On Fri, Oct 22, 2021 at 7:36 AM Helmut Jarausch <jarausch@××××××.be> wrote:
> >
> >
> > There are more than 55,000 files on some <PREVBackUp> which is located
> > on a BTRFS file system.
> > Standard 'rm -rf' is really slow.
> >
> > Is there anything I can do about this?
> >
>
> I don't have any solid suggestions as I haven't used btrfs in a while.
> File deletion speed is something that is very filesystem specific, but
> on most it tends to be slow.
>
> An obvious solution would be garbage collection, which is something
> used by some filesystems but I'm not aware of any mainstream ones.
> You can sort-of get that behavior by renaming a directory before
> deleting it.  Suppose you have a directory created by a build system
> and you want to do a new build.  Deleting the directory takes a long
> time.  So, first you rename it to something else (or move it someplace
> on the same filesystem which is fast), then you kick off your build
> which no longer sees the old directory, and then you can delete the
> old directory slowly at your leisure.  Of course, as with all garbage
> collection, you need to have the spare space to hold the data while it
> gets cleaned up.
>
> I'm not sure if btffs is any faster at deleting snapshots/reflinks
> than hard links.  I suspect it wouldn't be, but you could test that.
> Instead of populating a directory with hard links, create a snapshot
> of the directory tree, and then rsync over it/etc.  The result looks
> the same but is COW copies.  Again, I'm not sure that btrfs will be
> any faster at deleting reflinks than hard links though - they're both
> similar metadata operations.  I see there is a patch in the works for
> rsync that uses reflinks instead of hard links to do it all in one
> command.  That has a lot of benefits, but again I'm not sure if it
> will help with deletion.
>
> You could also explore other filesystems that may or may not have
> faster deletion, or look to see if there is any way to optimize it on
> btrfs.
>
> If you can spare the space, the option of moving the directory to make
> it look like it was deleted will work on basically any filesystem.  If
> you want to further automate it you could move it to a tmp directory
> on the same filesystem and have tmpreaper do your garbage collection.
> Consider using ionice to run it at a lower priority, but I'm not sure
> how much impact that has on metadata operations like deletion.
>
> --
> Rich
>