1 |
On Mon, Dec 20, 2021 at 1:52 PM Mark Knecht <markknecht@×××××.com> wrote: |
2 |
> |
3 |
> I've recently built 2 TrueNAS file servers. The first (and main) unit |
4 |
> runs all the time and serves to backup my home user machines. |
5 |
> Generally speaking I (currently) put data onto it using rsync but it |
6 |
> also has an NFS mount that serves as a location for my Raspberry Pi to |
7 |
> store duplicate copies of astrophotography pictures live as they come |
8 |
> off the DSLR in the middle of the night. |
9 |
> |
10 |
> ... |
11 |
> |
12 |
> The thing is that the ZIL is only used for synchronous writes and I |
13 |
> don't know whether anything I'm doing to back up my user machines, |
14 |
> which currently is just rsync commands, is synchronous or could be |
15 |
> made synchronous, and I do not know if the NFS writes from the R_Pi |
16 |
> are synchronous or could be made so. |
17 |
> |
18 |
|
19 |
Disclaimer: some of this stuff is a bit arcane and the documentation |
20 |
isn't very great, so I could be missing a nuance somewhere. |
21 |
|
22 |
First, one of your options is to set sync=always on the zfs dataset, |
23 |
if synchronous behavior is strongly desired. That will force ALL |
24 |
writes at the filesystem level to be synchronous. It will of course |
25 |
also normally kill performance but the ZIL may very well save you if |
26 |
your SSD performs adequately. This still only applies at the |
27 |
filesystem level, which may be an issue with NFS (read on). |
28 |
|
29 |
I'm not sure how exactly you're using rsync from the description above |
30 |
(rsyncd, directly client access, etc). In any case I don't think |
31 |
rsync has any kind of option to force synchronous behavior. I'm not |
32 |
sure if manually running a sync on the server after using rsync will |
33 |
use the ZIL or not. If you're using sync=always then that should |
34 |
cover rsync no matter how you're doing it. |
35 |
|
36 |
Nfs is a little different as both the server-side and client-side have |
37 |
possible asynchronous behavior. By default the nfs client is |
38 |
asynchronous, so caching can happen on the client before the file is |
39 |
even sent to the server. This can be disabled with the mount option |
40 |
sync on the client side. That will force all data to be sent to the |
41 |
server immediately. Any nfs server or filesystem settings on the |
42 |
server side will not have any impact if the client doesn't transmit |
43 |
the data to the server. The server also has a sync setting which |
44 |
defaults to on, and it additionally has another layer of caching on |
45 |
top of that which can be disabled with no_wdelay on the export. Those |
46 |
server-side settings probably delay anything getting to the filesystem |
47 |
and so they would have precedence over any filesystem-level settings. |
48 |
|
49 |
As you can see you need to use a bit of a kill-it-with-fire approach |
50 |
to get synchronous behavior, as it traditionally performs so poorly |
51 |
that everybody takes steps to try to prevent it from happening. |
52 |
|
53 |
I'll also note that the main thing synchronous behavior protects you |
54 |
from is unclean shutdown of the server. It has no bearing on what |
55 |
happens if a client goes down uncleanly. If you don't expect server |
56 |
crashes it may not provide much benefit. |
57 |
|
58 |
If you're using ZIL you should consider having the ZIL mirrored, as |
59 |
any loss of the ZIL devices will otherwise cause data loss. Use of |
60 |
the ZIL is also going to create wear on your SSD so consider that and |
61 |
your overall disk load before setting sync=always on the dataset. |
62 |
Since the setting is at the dataset level you could have multiple |
63 |
mountpoints and have a different sync policy for each. The default is |
64 |
normal POSIX behavior which only syncs when requested (sync, fsync, |
65 |
O_SYNC, etc). |
66 |
|
67 |
-- |
68 |
Rich |