Re: [gentoo-user] Synchronous writes over the network.

[Mark Knecht]

On Thu, Dec 23, 2021 at 10:27 AM Rich Freeman <rich0@g.o> wrote:
>
> On Thu, Dec 23, 2021 at 11:56 AM Mark Knecht <markknecht@×××××.com> wrote:
> >
<SNIP>
>
> > Instead
> > of a ZIL in machine 1 the SSD becomes a ZLOG cache most likely holding
> > a cached copy of the currently active astrophotography projects.
>
> I think you're talking about L2ARC.  I don't think "ZLOG" is a thing,
> and a log device in ZFS is just another name for ZIL (since that's
> what it is - a high performance data journal).
>

Thank you. Yes, L2ARC.

> L2ARC drives don't need to be mirrored and their failure is harmless.
> They generally only improve things, but of course they do nothing to
> improve write performance - just read performance.
>
> >    As always I'm interested in your comments about what works or
> > doesn't work about this sort of setup.
>
> Ultimately it all comes down to your requirements and how you use
> stuff.  What is the impact to you if you lose this real-time audio
> recording?  If you will just have to record something over again but
> that isn't a big deal, then what you're doing sounds fine to me.

Actually, no.

>   If
> you are recording stuff that is mission-critical and can't be repeated
> and you're going to lose a lot of money or reputation if you lose a
> recording, then I'd have that recording machine be pretty reliable
> which means redundant everything (server grade hardware with fault
> tolerance and RAID/etc, or split the recording onto two redundant sets
> of cheap consumer hardware).

Closer to mission critical.

When recording live music, most especially in situations with
lots of musicians, you don't want to miss a good take. In cases where
you are just capturing a band playing it's just about getting it on disk,
however in cases where you are adding to music that's already on disk,
say a vocalist singing live over the top of music the band played earlier
then having the hardware screw up a good take is really a downer.

>
> I do something similar - all the storage I care about is on
> Linux/ZFS/lizardfs with redundancy and backup.  I do process
> photos/video on a windows box on an NVMe, but that is almost never the
> only copy of my data.  I might offload media to the windows box from
> my camera, but if I lose that then I still have the camera.  I might
> do some processing on windows like generating thumbnails/etc on NVMe
> before I move it to network storage.  In the end though it goes to zfs
> on linux and gets backed up and so on.  If I need to process some
> videos I might copy data back to a windows NVMe for more performance
> if I don't want to directly spool stuff off the network, but my risks
> are pretty minimal if that goes down at any point.  And this is just
> personal stuff - I care about it and don't want to lose it, but it
> isn't going to damage my career if I lose it.  If I were dealing with
> data professionally it still wouldn't be a bad arrangement but I might
> invest in a few things differently.
>

In the case of recording audio it just gets down to how large a
project you are working on. 3 minute pop songs aren't much of an
issue. 10-20 stereo tracks at 96KHz isn't all that large. For those
the audio might fit in DRAM. However if you're working on some
wonderful 30 minute prog rock piece with 100 or more stereo tracks
it can get a lot larger but (in my mind anyway) the main desktop
machine will have some sort of M.2 and maybe it fits in there
and it gets read off hard disk before the session starts and there's
probably no problem.

I haven't given this a huge amount of worry because my current
machine does an almost perfect job with 8-9 year old technology.

In the case of astrophotography I will have multiple copies of the
original photos. The process of stacking the individual photos can
create gigabytes of intermediate files but as long as the originals
are safe then it's just a matter of starting over. In my astrophotography
setup I create about 50Mbyte per minute and take pictures for hours
so a set of photos coming in at 1-2GB and up to maybe 10GB isn't
uncommon. I might create 30-50GB of intermediate files which
eventually get deleted but they can reside on the server while I'm
working. None of that has to be terribly fast.

> Just ask yourself what hardware needs to fail for you to lose
> something you care about at any moment of time.  If you can tolerate
> the loss of just about any individual piece of hardware that's a
> pretty good first step for just about anything, and is really all you
> need for most consumer stuff.  Backups are fine as long as they're
> recent enough and you don't mind redoing work.
>
Agreed.

Thanks,
Mark