[gentoo-dev] Re: RFC: using .xz for doc/man/info compression

[Duncan]

Rich Freeman posted on Tue, 13 May 2014 08:18:25 -0400 as excerpted:

> Btrfs also supports file inlining, so every byte saved on small files
> does actually help (I believe the data structure that stores the inlined
> data doesn't have a fixed record size).

There's an option for it, altho I've not screwed with it and don't know 
the default without looking it up.

The overall metadata node size (set at mkfs.btrfs time) originally 
defaulted to the filesystem block size, which is the memory page size, 
thus 4096 bytes on x86/amd64 and I believe arm.  However, the metadata 
node size default recently changed to 16KiB (or page size where that is 
larger than 16KiB), altho I'd guess there's still more 4KiB node size 
users due to all the legacy btrfs out there, but 16KiB will certainly be 
the majority at some point.

Individual file inline size is certainly smaller than metadata node size, 
but again, I've not messed with that so don't know the actual default for 
it.

> Then again, btrfs also supports lzo compression and I believe this is
> fairly widely used, so I'm not sure that the impact of not compressing
> small files will be felt.

Of course there's gzip as well, and it's the (now legacy) default if 
compression is specified but not type, altho lzo is recommended as faster 
with "good enough" compression.

The other factor to consider is replication mode.  On a single device 
filesystem data replication mode is single by default, with metadata dup 
(two copies), except on detected ssd, where the metadata default is 
(somewhat controversially) single due to some ssds doing internal 
deduplication.  On multi-device filesystems the metadata default is (two-
copy, regardless of the number of devices) raid1, while the data default 
remains single.

So from a size perspective, assuming defaults of single data, dup or 
raid1 metadata, uncompressed, the cutover should be near 2048 bytes, 
since under that, duplicated metadata inlining will still be smaller than 
the 4096 byte data block size, while over that, sticking it in a single-
mode data extent should be more efficient.

Bottom line, there's enough btrfs variables including inlining size, data 
vs. metadata replication modes, metadata node sizes and compression and 
compression type, and the chances that gentoo btrfs users are likely to 
be tweaking at least one of those variables is high enough, that I'm not 
sure a generic ideal cutover makes a lot of sense, but to the extent that 
there is one, it's likely to be near 2048 bytes.

FWIW I believe I'm still using portage bzip2 docs compression by default 
here, altho in the context of this thread I should really examine that 
since I use compress=lzo at the filesystem level.  Both data and metadata 
are raid1 here, so inlining doesn't matter except that AFAIK inlining is 
NOT compressed while data extents can be, so portage level compression is 
likely to make even less difference if it's in the range that portage 
level bzip2 compression makes it small enough to be inlined, vs not 
portage level compressed but then big enough to not be inlined, thus 
btrfs-level transparent lzo compressed as a data extent.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman