[gentoo-dev] Change in Gentoo ZFS packaging policy

[Richard Yao]

I have had ClusterHQ's partial stable ABI in Gentoo's packaging since
December (which helped to shake out bugs) and have just done another
update of it with various bug fixes, including many from HEAD.

Some might notice that merging a pull request that isn't fully reviewed
is a fairly big departure from the rigorous vetting that I did on
patches to ensure that every update was absolutely bug free. I had a
good span of about 3 years without Gentoo specific regressions that I
can recall.

While I did my best to avoid introducing regressions in Gentoo's ZoL
packaging, the /dev/zfs ABI has been unstable. That allowed for
potential upgrade pains such as 0.6.2 -> 0.6.3 where those with
mismatched modules and userspace code had problems. All ZoL packaging in
Gentoo has been marked as "testing" because of it. The partial stable
ABI presents offers an opportunity to change this such that users will
have the option of either stable packages (well tested and vetted) or
testing packages (vetted by me and pushed out for the community to vet
further).

In specific, the partial stable ABI should allow the following functions
to work whenever mismatches occur, provided that both the userland and
kernel code are mismatched using versions after the transition has taken
place:

- zpool import
- zpool iostat
- zpool tryimport
- zpool export
- zpool list
- zpool get
- zpool status
- zfs mount
- zfs umount
- zfs clone
- zfs snapshot
- zfs rollback
- zfs destroy
- zfs create
- zfs list
- zfs get
- zfs set
- zfs inherit
- zfs send
- zfs recv (part of stable API, but not ABI in the current code, plan to
resolve in the future)
- zfs bookmark
- zfs hold
- zfs release

It also provides convenient functions to control these operations
through libzfs_core. The man pages are a draft from a few months ago,
but include all of the non-zpool commands. This will be rectified in the
near future so that all functions are thoroughly documented.

After consulting with a few others, I decided in December to transition
Gentoo to this now in the belief that the long term benefits outweigh
the short term pain of the transition. Specifically, I want to exchange
all of the pain the /dev/zfs ABI being unstable will cause over years
for a brief period of pain now.

Having any pain at all is not ideal, but this is the first time we have
had a plan to ensure that this will be the last time (for the subset I
mentioned, things outside of it will likely be added in the future based
on need/importance).

Doing an update today to fix the regressions that were reported has
given me the opportunity to reflect on how I have handled things since
the December update. At present, I am working on multiple things and
while it is hard to keep track of every issue, I would like to minimize
the amount of pain that occurs during this transition. My plan is to
complete the transition by marking things stable 2 weeks after I stop
hearing reports of regressions.

If there are any regressions I have missed, I would like to hear about
them. Please contact me in #zfsonlinux on the freenode IRC network so
that I can resolve it ASAP. That way the regression can cut in line in
my queue and be handled faster than the regressions I missed prior to
the December code push.

Having said that, those on other distributions are welcome to the code
and to contact me about issues (please ping through freenode for the
fastest response). It can be found on github:

https://github.com/ryao/zfs/commits/gentoo-zfs-0.6.5.3-release
https://github.com/ryao/spl/commits/gentoo-spl-0.6.5.3-release

Tarballs of patches can be found here:

http://dev.gentoo.org/~ryao/dist/zfs-0.6.5.3-patches-p1.tar.xz
SHA256: 765a66adf67d0a3ae6a699561b98a5158d464e3b6ed413a72bdbbe6e6252ba66
SHA1: 56079ecd2543c44191cd747ce28a28a2bab8c064
MD5: 0a92dae487f507616e0850ee9760932f

http://dev.gentoo.org/~ryao/dist/spl-0.6.5.3-patches-p0.tar.xz
SHA256: 8e652d41eba421720bcecee99077d3f3c375153809426011f04a2c64aa181ca7
SHA1: 50cf449d7ba1772b967446bad7c12ac29fee7b6c
MD5: 6d3c2fe3bfff26f575120aac623f8dda

The ZFS shortlog:

AndCycle (1):
      Obey arc_meta_limit default size when changing arc_max

Benjamin Albrecht (1):
      Activate LVM volume groups before looking for zpools.

Brian Behlendorf (15):
      Fix maybe uninitialized
      Follow 0/-E convention for module load errors
      Fix --enable-linux-builtin
      Use large stacks when available
      Either _ILP32 or _LP64 must be defined
      Fix zfsctl_lookup_objset() deadlock
      Change zfs_snapshot_lock from mutex to rw lock
      Set 'zfs_expire_snapshot=0' to disable auto-unmount
      Hold the zfs_snapentry_t before dispatch
      Handle block pointers with a corrupt logical size
      Handle damaged blk_birth in dsl_deadlist_insert()
      Fix vdev_queue_aggregate() deadlock
      Fix zfs_vdev_aggregation_limit bounds checking
      Fix ztest truncated cache file
      Fix z_xattr_lock/z_teardown_lock inversion

Chunwei Chen (11):
      Fix fail path in zfs_znode_alloc
      Fix snapshot automount behavior when concurrent or fail
      Linux 4.4 compat: make_request_fn returns blk_qc_t
      Linux 4.4 compat: xattr operations takes xattr_handler
      Fix zdb calling behavior in ztest
      Fix zdb_dump_block on little endian systems
      Use spa as key besides objsetid for snapentry
      Fix snapshot automount race cause EREMOTE
      Fix uio_prefaultpages for 0 length iovec
      Fix empty xattr dir causing lockup
      Make xattr dir truncate and remove in one tx

DHE (1):
      libzfs: handle EDOM errors

Jason Zaman (1):
      sysmacros: Make P2ROUNDUP not trigger int overflow

Kamil Domański (1):
      Skip GPL-only symbols test when cross-compiling

Ned Bass (1):
      Prevent SA length overflow

Olaf Faaland (1):
      Remove "index" column from dbufstat.py

Richard Yao (5):
      ClusterHQ Proposed API extensions
      Only trigger SET_ERROR tracepoint event on error
      Unconditionally build zdb and ztest with -DDEBUG
      Revert "Switch ztest mmap(2) ASSERTs to VERIFYs"
      Refresh ClusterHQ Proposed API extensions against ZoL PR

Turbo Fredriksson (1):
      Unmount is part of the shutdown process, not the boot process.

tuxoko (3):
      Fix null pointer in arc_kmem_reap_now on 32-bit
      Fix zfs_dirty_data_max overflow on 32-bit
      Prevent rm modules.* when make install

The SPL shortlog:

Brian Behlendorf (1):
      Fix --enable-linux-builtin

Chunwei Chen (1):
      Don't call kmem_cache_shrink from shrinker

Dimitri John Ledkov (1):
      spl-kmem-cache: include linux/prefetch.h for prefetchw()

Jason Zaman (1):
      sysmacros: Make P2ROUNDUP not trigger int overflow

Richard Yao (2):
      Fix race between getf() and areleasef()
      Make taskq_member() use ->journal_info

tuxoko (2):
      Fix taskq dynamic spawning
      Prevent rm modules.* when make install