Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format

[Michał Górny]

On Sat, 2018-11-10 at 09:37 -0500, Alec Warner wrote:
> On Sat, Nov 10, 2018 at 8:09 AM Michał Górny <mgorny@g.o> wrote:
>
> > Hi, everyone.
> >
> > The Gentoo's tbz2/xpak package format is quite old.  We've made a few
> > incompatible changes in the past (most notably, allowing non-bzip2
> > compression and multi-instance naming) but the core design stayed
> > the same.  I think we should consider changing it, for the reasons
> > outlined below.
> >
> > The rough format description can be found in xpak(5).  Basically, it's
> > a regular compressed tarball with binary metadata blob appended
> > to the end.  As such, it looks like a regular compressed tarball
> > to the compression tools (with some ignored junk at the end).
> > The metadata is entirely custom format and needs dedicated tools
> > to manipulate.
> >
> >
> > The current format has a few advantages whose preserving would probably
> > be worthwhile:
> >
> > + The binary package is a single flat file.
> >
> > + It is reasonably compatible with regular compressed tarball,
> > so the users can unpack it using standard tools (except for metadata).
> >
> > + The metadata is uncompressed and can be quickly found without touching
> > the compressed data.
> >
> > + The metadata can be updated (e.g. as result of pkgmove) without
> > touching the compressed data.
> >
> >
> > However, it has a few disadvantages as well:
> >
> > - The metadata is entirely custom binary format, requiring dedicated
> > tools to read or edit.
> >
> > - The metadata format is relying on customary behavior of compression
> > tools that ignore junk following the compressed data.
> >
>
> I agree this is a problem in theory, but I haven't seen it as a problem in
> practice. Have you observed any problems around this setup?

Historically one of the parallel compressor variants did not support
this.

> >
> > - By placing the metadata at the end of file, we make it rather hard to
> > read the metadata from remote location (via FTP, HTTP) without fetching
> > the whole file.  [NB: it's technically possible but probably not worth
> > the effort]
>
>
> > - By requiring the custom format to be at the end of file, we make it
> > impossible to trivially cover it with a OpenPGP signature without
> > introducing another custom format.
> >
>
> Its trivial to cover with a detached sig, no?
>
>
> >
> > - While the format might allow for some extensibility, it's rather
> > evolutionary dead end.
> >
>
> I'm not even sure how to quantify this, it just sounds like your subjective
> opinion (which is fine, but its not factual.)
>
>
> >
> >
> > I think the key points of the new format should be:
> >
> > 1. It should reuse common file formats as much as possible, with
> > inventing as little custom code as possible.
> >
> > 2. It should allow for easy introspection and editing by users without
> > dedicated tools.
> >
>
> So I'm less confident in the editing use cases; do users edit their binpkgs
> on a regular basis?

It's useful for debugging stuff.  I had to use hexedit on xpak
in the past.  Believe me, it's nowhere close to pleasant.

> > 3. The metadata should allow for lookup without fetching the whole
> > binary package.
> >
> > 4. The format should allow for some extensions without having to
> > reinvent the wheel every time.
> >
> > 5. It would be nice to preserve the existing advantages.
> >
> >
> > My proposal
> > ===========
> >
> > Basic format
> > ------------
> > The base of the format is a regular compressed tarball.  There's no junk
> > appended to it but the metadata is stored inside it as
> > /var/db/pkg/${PF}.  The contents are as compatible with the actual vdb
> > format as possible.
> >
>
> Just to clarify, you are suggesting we store the metadata inside the
> contents of the binary package itself (e.g. where the other files that get
> merged to the liveFS are?) What about collisions?
>
> E.g. I install 'machine-images/gentoo-disk-image-1.2.3' on a machine that
> already has 'machine-images/gentoo-disk-image-1.2.3' installed, won't it
> overwrite files in the VDB at qmerge time?

Portage will obviously move the files out, and process them as metadata.
 The idea is to precisely use a directory that can't be normally part
of binary packages, so can't cause collisions with real files (even if
they're very unlikely to ever happen).

> > This has the following advantages:
> >
> > + Binary package is still stored as a single file.
> >
> > + It uses a standard compressed .tar format, with minimal customization.
> >
> > + The user can easily inspect and modify the packages with standard
> > tools (tar and the compressor).
> >
> > + If we can maintain reasonable level of vdb compatibility, the user can
> > even emergency-install a package without causing too much hassle (as it
> > will be recorded in vdb); ideally Portage would detect this vdb entry
> > and support fixing the install afterwards.
> >
>
> I'm not certain this is really desired.

Are you saying it's better that user emergency-installs a package
without recording it in vdb, and ends up with mess of collisions
and untracked files?

Just because you don't like some use case doesn't mean it's not gonna
happen.  Either you prepare for it and make the best of it, or you
pretend it's not gonna happen and cause extra pain to users.

> > Optimizing for easy recognition
> > -------------------------------
> > In order to make it possible for magic-based tools such as file(1) to
> > easily distinguish Gentoo binary packages from regular tarballs, we
> > could (ab)use the volume label field, e.g. use:
> >
> >   $ tar -V 'gpkg: app-foo/bar-1' -c ...
> >
> > This will add a volume label as the first file entry inside the tarball,
> > which does not affect extracting but can be trivially matched via magic
> > rules.
> >
> > Note: this is meant to be used as a method for fast binary package
> > recognition; I don't think we should reject (hand-modified) binary
> > packages that lack this label.
> >
> >
> > Optimizing for metadata reading/manipulation performance
> > --------------------------------------------------------
> > The main problem with using a single tarball for both metadata and data
> > is that normally you'd have to decompress everything to reliably unpack
> > metadata, and recompress everything to update it.  This problem can be
> > addressed by a few optimization tricks.
> >
>
> These performance goals seem a little bit ill defined.
>
> 1) Where are users reporting slowness in binpkg operations?
> 2) What is the cause of the slowness?

Those are optimizations to not cause slowness compared to the current
format.  Main use case is recreating package index which would require
rereading the metadata of all binary packages.

> Like I could easily see a potential user with many large binpkgs, and the
> current implementation causing them issues because
> they have to decompress and seek a bunch to read the metadata out of their
> 1.2GB binpkg. But i'm pretty sure this isn't most users.
>
>
> >
> > Firstly, all metadata files are packed to the archive before data files.
> >  With a slightly customized unpacker, we can stop decompressing as soon
> > as we're past metadata and avoid decompressing the whole archive.  This
> > will also make it possible to read metadata from remote files without
> > fetching far past the compressed metadata block.
> >
>
> So this seems to basically go against your goals of simple common tooling?

No.  My goal is to make it compatible with simple common tooling.  You
can still use the simple tooling to read/write them.  The optimized
tools are only needed to efficiently handle special use cases.

> > Secondly, if we're up for some more tricks, we could technically split
> > the tarball into metadata and data blocks compressed separately.  This
> > will need a bit of archiver customization but it will make it possible
> > to decompress the metadata part without even touching compressed data,
> > and to replace it without recompressing data.
> >
> > What's important is that both tricks proposed maintain backwards
> > compatibility with regular compressed tarballs.  That is, the user will
> > still be able to extract it with regular archiving tools.
>
>
> So my recollection is that debian uses common format AR files for the main
> deb.
> Then they have 2 compressed tarballs, one for metadata, and one for data.
>
> This format seems to jive with many of your requirements:
>
>  - 'ar' can retrieve individual files from the archive.
>  - The deb file itself is not compressed, but the tarballs inside *are*
> compressed.
>  - The metadata and data are compressed separately.
>  - Anyone can edit this with normal tooling (ar, tar)
>
> In short; why should we event a new format?

Because nobody knows how to use 'ar', compared to how almost every
Gentoo user can use 'tar' immediately?  Of course we could alternatively
just use a nested tarball but I wanted to keep the possibility
of actually being able to 'tar -xf' it without having to extract nested
archives.

-- 
Best regards,
Michał Górny