Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format

[Zac Medico]

On 11/10/2018 06:37 AM, Alec Warner wrote:
>
> On Sat, Nov 10, 2018 at 8:09 AM Michał Górny <mgorny@g.o
> <mailto:mgorny@g.o>> wrote:
>
>     Hi, everyone.
>
>     The Gentoo's tbz2/xpak package format is quite old.  We've made a few
>     incompatible changes in the past (most notably, allowing non-bzip2
>     compression and multi-instance naming) but the core design stayed
>     the same.  I think we should consider changing it, for the reasons
>     outlined below.
>
>     The rough format description can be found in xpak(5).  Basically, it's
>     a regular compressed tarball with binary metadata blob appended
>     to the end.  As such, it looks like a regular compressed tarball
>     to the compression tools (with some ignored junk at the end).
>     The metadata is entirely custom format and needs dedicated tools
>     to manipulate.
>
>
>     The current format has a few advantages whose preserving would probably
>     be worthwhile:
>
>     + The binary package is a single flat file.
>
>     + It is reasonably compatible with regular compressed tarball,
>     so the users can unpack it using standard tools (except for metadata).
>
>     + The metadata is uncompressed and can be quickly found without touching
>     the compressed data.
>
>     + The metadata can be updated (e.g. as result of pkgmove) without
>     touching the compressed data.
>
>
>     However, it has a few disadvantages as well:
>
>     - The metadata is entirely custom binary format, requiring dedicated
>     tools to read or edit.
>
>     - The metadata format is relying on customary behavior of compression
>     tools that ignore junk following the compressed data.
>
>
> I agree this is a problem in theory, but I haven't seen it as a problem
> in practice. Have you observed any problems around this setup?

In portage we use head -c to selected the compressed data, since zstd
doesn't handle the xpak trailer well.

>
>     - By placing the metadata at the end of file, we make it rather hard to
>     read the metadata from remote location (via FTP, HTTP) without fetching
>     the whole file.  [NB: it's technically possible but probably not worth
>     the effort] 
>
>
>     - By requiring the custom format to be at the end of file, we make it
>     impossible to trivially cover it with a OpenPGP signature without
>     introducing another custom format.
>
>
> Its trivial to cover with a detached sig, no?
>  
>
>
>     - While the format might allow for some extensibility, it's rather
>     evolutionary dead end.
>
>
> I'm not even sure how to quantify this, it just sounds like your
> subjective opinion (which is fine, but its not factual.)

Yeah the xpak trailer is flexible enough, but I'm not opposed to
supporting a different format.

>
>     I think the key points of the new format should be:
>
>     1. It should reuse common file formats as much as possible, with
>     inventing as little custom code as possible.
>
>     2. It should allow for easy introspection and editing by users without
>     dedicated tools.
>
>
> So I'm less confident in the editing use cases; do users edit their
> binpkgs on a regular basis?

Yes, gentoo/profiles/updates package renames an slot moves are a form of
this.

>
>     3. The metadata should allow for lookup without fetching the whole
>     binary package.
>
>     4. The format should allow for some extensions without having to
>     reinvent the wheel every time.
>
>     5. It would be nice to preserve the existing advantages.
>
>
>     My proposal
>     ===========
>
>     Basic format
>     ------------
>     The base of the format is a regular compressed tarball.  There's no junk
>     appended to it but the metadata is stored inside it as
>     /var/db/pkg/${PF}.  The contents are as compatible with the actual vdb
>     format as possible.
>
>
> Just to clarify, you are suggesting we store the metadata inside the
> contents of the binary package itself (e.g. where the other files that
> get merged to the liveFS are?) What about collisions?
>
> E.g. I install 'machine-images/gentoo-disk-image-1.2.3' on a machine
> that already has 'machine-images/gentoo-disk-image-1.2.3' installed,
> won't it overwrite files in the VDB at qmerge time?

I haven't looked into it but maybe we can use "nil control directory
names" to embed things, like http://savannah.gnu.org/projects/swbis
claims to use.

>     This has the following advantages:
>
>     + Binary package is still stored as a single file.
>
>     + It uses a standard compressed .tar format, with minimal customization.
>
>     + The user can easily inspect and modify the packages with standard
>     tools (tar and the compressor).
>
>     + If we can maintain reasonable level of vdb compatibility, the user can
>     even emergency-install a package without causing too much hassle (as it
>     will be recorded in vdb); ideally Portage would detect this vdb entry
>     and support fixing the install afterwards.
>
>
> I'm not certain this is really desired.

Yeah I don't like it either, I'd prefer to keep the metadata someplace
where it can't overwrite files in the installed package database.

>
>     Optimizing for easy recognition
>     -------------------------------
>     In order to make it possible for magic-based tools such as file(1) to
>     easily distinguish Gentoo binary packages from regular tarballs, we
>     could (ab)use the volume label field, e.g. use:
>
>       $ tar -V 'gpkg: app-foo/bar-1' -c ...
>
>     This will add a volume label as the first file entry inside the tarball,
>     which does not affect extracting but can be trivially matched via magic
>     rules.
>
>     Note: this is meant to be used as a method for fast binary package
>     recognition; I don't think we should reject (hand-modified) binary
>     packages that lack this label.
>
>
>     Optimizing for metadata reading/manipulation performance
>     --------------------------------------------------------
>     The main problem with using a single tarball for both metadata and data
>     is that normally you'd have to decompress everything to reliably unpack
>     metadata, and recompress everything to update it.  This problem can be
>     addressed by a few optimization tricks.
>
>
> These performance goals seem a little bit ill defined.
>
> 1) Where are users reporting slowness in binpkg operations?
> 2) What is the cause of the slowness?

Yeah I'd like more information here too.

> Like I could easily see a potential user with many large binpkgs, and
> the current implementation causing them issues because
> they have to decompress and seek a bunch to read the metadata out of
> their 1.2GB binpkg. But i'm pretty sure this isn't most users.
>  
>
>
>     Firstly, all metadata files are packed to the archive before data files.
>      With a slightly customized unpacker, we can stop decompressing as soon
>     as we're past metadata and avoid decompressing the whole archive.  This
>     will also make it possible to read metadata from remote files without
>     fetching far past the compressed metadata block.
>
>
> So this seems to basically go against your goals of simple common tooling?
>  
>
>
>     Secondly, if we're up for some more tricks, we could technically split
>     the tarball into metadata and data blocks compressed separately.  This
>     will need a bit of archiver customization but it will make it possible
>     to decompress the metadata part without even touching compressed data,
>     and to replace it without recompressing data.
>
>     What's important is that both tricks proposed maintain backwards
>     compatibility with regular compressed tarballs.  That is, the user will
>     still be able to extract it with regular archiving tools.
>
>
> So my recollection is that debian uses common format AR files for the
> main deb.
> Then they have 2 compressed tarballs, one for metadata, and one for data.
>
> This format seems to jive with many of your requirements:
>
>  - 'ar' can retrieve individual files from the archive.
>  - The deb file itself is not compressed, but the tarballs inside *are*
> compressed.
>  - The metadata and data are compressed separately.
>  - Anyone can edit this with normal tooling (ar, tar)
>
> In short; why should we event a new format?

Maybe we can borrow some ideas from
http://savannah.gnu.org/projects/swbis which claims to be capable of
creating and verifying a tarball with GPG signatures embedded in the
tarball.

>
>     Adding OpenPGP signatures
>     -------------------------
>     This is the main XXX here.
>
>     Technically, the most obvious solution is to cover the entire tarball
>     with OpenPGP signature.  However, this has the disadvantage that
>     the verification requires fetching the whole file.
>
>     I will look into possibility of having partial signatures.
>
>
>     --
>     Best regards,
>     Michał Górny
>


-- 
Thanks,
Zac