Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Stefan G. Weichinger]

Am 27.12.2012 01:18, schrieb Alan McKinnon:

> I am *very* impressed with ZFS for this. Yes, I know, it's not really
> there on Linux - I use it on FreeBSD (FreeNAS).
>
> It has everything I've wanted in a filesystem for a long time, and all
> the crap I've stuffed into my head over many years related to storage
> just goes away. It doesn't go to some place I don't have to deal with
> it, it just ceases to exist. Very nice.
>
> There is no more weird partitions from the days of DOS, no PV/VG/LV to
> remember the details of. There is only storage and ZFS knows what I
> want to happen with each "chunk" of it. A "chunk" (my term) in this
> context is a directory and everything below it.
>
> ZFS doesn't have partitions and filesystems. It has volumes. A volume
> is sort of a cross between a filesystem (you mount it and can assign
> quotas to it) and a directory (you assign permissions and ownerships to
> it). You can overcommit storage space and quotas - you do not get "disk
> full" errors and three days of nightmares while you figure out how to
> deal with this. the FS just tells you it used more than the allocated
> space and keeps telling you till you get it under the limit.
>
> mv'ing a few TB of video to a different FS to free up space is not fun
> at all, but with ZFS it's like an mv on the same FS (that volume thing
> again). It checksums every write and lets you know if things fail. It
> has proper snapshots built in - that's proper as in copy-on-write so
> they don't really take up space until you start modifying files. Your
> media collection is like mine - I only add to it and seldom delete, so
> I have months of snapshots that consume about 1% extras space. Dale's
> rm problem cannot happen to me anymore hehehehe ;-)
>
> In summary, it does everything I want and does it well. It can also do
> other things I don't want but others might (eg de-dupe).

I also used ZFS for that kind of storage and it was very pleasant to
work with. Performance is an issue if you use zfs-fuse for example,
although there are people running zfs-on-linux on LUKS for their main
working machine (using an SSD, that helps!):

https://mthode.org/

describes such a setup (although slightly off-topic in context of media
libraries).

I had the pleasure to write a feature about that and provide a demo-vm:

http://www.oops.co.at/en/publications/beitrag-fuer-linux-magazin-012013-zfs-und-luks

(german feature, sorry ...)

-

Using snapshots with ZFS really is fun and let's you rethink stuff. It
makes doing backups easier and you can send and receive them via pipes
(think ssh here).

ZFS thinks of storage in the way we think of RAM: plug in some more and
everything (as in "every filesystem") is able to use it, just a pool of
ressources (actually it uses that term all over, "a zfs pool called tank").

I'd love to use it as root-fs sometimes, but I still hesitate. Might be
better to stay with the filesystems most linux-users use, just to
benefit of the huge tester-group :-)

For media-storage I wouldn't hesitate to run zfs-on-linux on a mirror of
2 disks or so. With that you benefit of the so-called self-healing:

If the checksum for block X on disk sda isn't correct, very likely the
checksum for block X on disk sdb is still valid. ZFS sees that, creates
a new block X on sda, with the correct content&checksum and drops the
corrupted block (and doesn't reuse it, AFAIK).

That is a great feature ... maybe not too important for video where the
occasional bit error isn't that much of a problem. But good to have for
other stuff ...

Greets, Stefan