Re: [gentoo-user] Synchronous writes over the network.

[Rich Freeman]

On Mon, Dec 20, 2021 at 1:52 PM Mark Knecht <markknecht@×××××.com> wrote:
>
> I've recently built 2 TrueNAS file servers. The first (and main) unit
> runs all the time and serves to backup my home user machines.
> Generally speaking I (currently) put data onto it using rsync but it
> also has an NFS mount that serves as a location for my Raspberry Pi to
> store duplicate copies of astrophotography pictures live as they come
> off the DSLR in the middle of the night.
>
> ...
>
> The thing is that the ZIL is only used for synchronous writes and I
> don't know whether anything I'm doing to back up my user machines,
> which currently is just rsync commands, is synchronous or could be
> made synchronous, and I do not know if the NFS writes from the R_Pi
> are synchronous or could be made so.
>

Disclaimer: some of this stuff is a bit arcane and the documentation
isn't very great, so I could be missing a nuance somewhere.

First, one of your options is to set sync=always on the zfs dataset,
if synchronous behavior is strongly desired.  That will force ALL
writes at the filesystem level to be synchronous.  It will of course
also normally kill performance but the ZIL may very well save you if
your SSD performs adequately.  This still only applies at the
filesystem level, which may be an issue with NFS (read on).

I'm not sure how exactly you're using rsync from the description above
(rsyncd, directly client access, etc).  In any case I don't think
rsync has any kind of option to force synchronous behavior.  I'm not
sure if manually running a sync on the server after using rsync will
use the ZIL or not.  If you're using sync=always then that should
cover rsync no matter how you're doing it.

Nfs is a little different as both the server-side and client-side have
possible asynchronous behavior.  By default the nfs client is
asynchronous, so caching can happen on the client before the file is
even sent to the server.  This can be disabled with the mount option
sync on the client side.  That will force all data to be sent to the
server immediately.  Any nfs server or filesystem settings on the
server side will not have any impact if the client doesn't transmit
the data to the server.  The server also has a sync setting which
defaults to on, and it additionally has another layer of caching on
top of that which can be disabled with no_wdelay on the export.  Those
server-side settings probably delay anything getting to the filesystem
and so they would have precedence over any filesystem-level settings.

As you can see you need to use a bit of a kill-it-with-fire approach
to get synchronous behavior, as it traditionally performs so poorly
that everybody takes steps to try to prevent it from happening.

I'll also note that the main thing synchronous behavior protects you
from is unclean shutdown of the server.  It has no bearing on what
happens if a client goes down uncleanly.  If you don't expect server
crashes it may not provide much benefit.

If you're using ZIL you should consider having the ZIL mirrored, as
any loss of the ZIL devices will otherwise cause data loss.  Use of
the ZIL is also going to create wear on your SSD so consider that and
your overall disk load before setting sync=always on the dataset.
Since the setting is at the dataset level you could have multiple
mountpoints and have a different sync policy for each.  The default is
normal POSIX behavior which only syncs when requested (sync, fsync,
O_SYNC, etc).

-- 
Rich