Gentoo Archives: gentoo-user

From: Helmut Jarausch <jarausch@××××××.be>
To: gentoo-user@l.g.o
Subject: Re: [gentoo-user] [OT] how to delete a directory tree really fast
Date: Fri, 22 Oct 2021 19:21:21
Message-Id: U3VALK7D.LGI5DDKA.UOW575HY@43S3X6C6.DWSQN6JG.N3YRUXDL
In Reply to: Re: [gentoo-user] [OT] how to delete a directory tree really fast by Vitor Hugo Nunes dos Santos
1 On 10/22/2021 06:15:58 PM, Vitor Hugo Nunes dos Santos wrote:
2 > The real solution would have been having a subvolume for the
3 > directory.
4 > Subvolume deletion on BTRFS is near instant.
5 > Same for ZFS with datasets, etc.
6
7 Thanks!
8 Is it possible to have a hard link from one subvolume to a different
9 one?
10
11 >
12 > October 22, 2021 9:50 AM, "Rich Freeman" <rich0@g.o> wrote:
13 >
14 > > On Fri, Oct 22, 2021 at 8:39 AM Miles Malone
15 > > <m.malone@××××××××××××××××××.net> wrote:
16 > >
17 > >> small files... (Certainly dont quote me here, but wasnt JFS the
18 > king
19 > >> of that back in the day? I cant quite recall)
20 > >
21 > > It is lightning fast on lizardfs due to garbage collection, but
22 > > metadata on lizardfs is expensive, requiring RAM on the master
23 > server
24 > > for every inode. I'd never use it for lots of small files.
25 > >
26 > > My lizardfs master is using 609MiB for 1,111,394 files (the bulk of
27 > > which are in snapshots, which create records for every file inside,
28 > so
29 > > if you snapshot 100k files you end up with 200k files). Figure 1kB
30 > > per file to be safe. Not a big deal if you're storing large files
31 > > (which is what I'm mostly doing). Performance isn't eye-popping
32 > > either - I have no idea how well it would work for something like a
33 > > build system where IOPS matters. For bulk storage of big stuff
34 > though
35 > > it is spectacular, and scales very well.
36 > >
37 > > Cephfs also uses delayed deletion. I have no idea how well it
38 > > performs, or what the cost of metadata is, though I suspect it is a
39 > > lot smarter about RAM requirements on the metadata server. Well,
40 > > maybe, at least in the past it wasn't all that smart about RAM
41 > > requirements on the object storage daemons. I'd seriously look at it
42 > > if doing anything new.
43 > >
44 > > Distributed filesystems tend to be garbage collected simply due to
45 > > latency. There are data integrity benefits to synchronous writes,
46 > but
47 > > there is rarely much benefit on blocking on delections, so why do
48 > it?
49 > > These filesystems already need all kinds of synchronization
50 > > capabilities due to node failures, so syncing deletions is just a
51 > > logical design.
52 > >
53 > > For conventional filesystems a log-based filesystem is naturally
54 > > garbage-collected, but those can have their own issues.
55 > >
56 > > --
57 > > Rich
58 >
59 >
60 >
61 >

Replies