Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format

[Alec Warner]

On Sun, Nov 11, 2018 at 3:29 AM Michał Górny <mgorny@g.o> wrote:

> On Sat, 2018-11-10 at 09:37 -0500, Alec Warner wrote:
> > On Sat, Nov 10, 2018 at 8:09 AM Michał Górny <mgorny@g.o> wrote:
> >
> > > Hi, everyone.
> > >
> > > The Gentoo's tbz2/xpak package format is quite old.  We've made a few
> > > incompatible changes in the past (most notably, allowing non-bzip2
> > > compression and multi-instance naming) but the core design stayed
> > > the same.  I think we should consider changing it, for the reasons
> > > outlined below.
> > >
> > > The rough format description can be found in xpak(5).  Basically, it's
> > > a regular compressed tarball with binary metadata blob appended
> > > to the end.  As such, it looks like a regular compressed tarball
> > > to the compression tools (with some ignored junk at the end).
> > > The metadata is entirely custom format and needs dedicated tools
> > > to manipulate.
> > >
> > >
> > > The current format has a few advantages whose preserving would probably
> > > be worthwhile:
> > >
> > > + The binary package is a single flat file.
> > >
> > > + It is reasonably compatible with regular compressed tarball,
> > > so the users can unpack it using standard tools (except for metadata).
> > >
> > > + The metadata is uncompressed and can be quickly found without
> touching
> > > the compressed data.
> > >
> > > + The metadata can be updated (e.g. as result of pkgmove) without
> > > touching the compressed data.
> > >
> > >
> > > However, it has a few disadvantages as well:
> > >
> > > - The metadata is entirely custom binary format, requiring dedicated
> > > tools to read or edit.
> > >
> > > - The metadata format is relying on customary behavior of compression
> > > tools that ignore junk following the compressed data.
> > >
> >
> > I agree this is a problem in theory, but I haven't seen it as a problem
> in
> > practice. Have you observed any problems around this setup?
>
> Historically one of the parallel compressor variants did not support
> this.
>
> > >
> > > - By placing the metadata at the end of file, we make it rather hard to
> > > read the metadata from remote location (via FTP, HTTP) without fetching
> > > the whole file.  [NB: it's technically possible but probably not worth
> > > the effort]
> >
> >
> > > - By requiring the custom format to be at the end of file, we make it
> > > impossible to trivially cover it with a OpenPGP signature without
> > > introducing another custom format.
> > >
> >
> > Its trivial to cover with a detached sig, no?
> >
> >
> > >
> > > - While the format might allow for some extensibility, it's rather
> > > evolutionary dead end.
> > >
> >
> > I'm not even sure how to quantify this, it just sounds like your
> subjective
> > opinion (which is fine, but its not factual.)
> >
> >
> > >
> > >
> > > I think the key points of the new format should be:
> > >
> > > 1. It should reuse common file formats as much as possible, with
> > > inventing as little custom code as possible.
> > >
> > > 2. It should allow for easy introspection and editing by users without
> > > dedicated tools.
> > >
> >
> > So I'm less confident in the editing use cases; do users edit their
> binpkgs
> > on a regular basis?
>
> It's useful for debugging stuff.  I had to use hexedit on xpak
> in the past.  Believe me, it's nowhere close to pleasant.
>

> > > 3. The metadata should allow for lookup without fetching the whole
> > > binary package.
> > >
> > > 4. The format should allow for some extensions without having to
> > > reinvent the wheel every time.
> > >
> > > 5. It would be nice to preserve the existing advantages.
> > >
> > >
> > > My proposal
> > > ===========
> > >
> > > Basic format
> > > ------------
> > > The base of the format is a regular compressed tarball.  There's no
> junk
> > > appended to it but the metadata is stored inside it as
> > > /var/db/pkg/${PF}.  The contents are as compatible with the actual vdb
> > > format as possible.
> > >
> >
> > Just to clarify, you are suggesting we store the metadata inside the
> > contents of the binary package itself (e.g. where the other files that
> get
> > merged to the liveFS are?) What about collisions?
> >
> > E.g. I install 'machine-images/gentoo-disk-image-1.2.3' on a machine that
> > already has 'machine-images/gentoo-disk-image-1.2.3' installed, won't it
> > overwrite files in the VDB at qmerge time?
>
> Portage will obviously move the files out, and process them as metadata.
>  The idea is to precisely use a directory that can't be normally part
> of binary packages, so can't cause collisions with real files (even if
> they're very unlikely to ever happen).
>
> > > This has the following advantages:
> > >
> > > + Binary package is still stored as a single file.
> > >
> > > + It uses a standard compressed .tar format, with minimal
> customization.
> > >
> > > + The user can easily inspect and modify the packages with standard
> > > tools (tar and the compressor).
> > >
> > > + If we can maintain reasonable level of vdb compatibility, the user
> can
> > > even emergency-install a package without causing too much hassle (as it
> > > will be recorded in vdb); ideally Portage would detect this vdb entry
> > > and support fixing the install afterwards.
> > >
> >
> > I'm not certain this is really desired.
>
> Are you saying it's better that user emergency-installs a package
> without recording it in vdb, and ends up with mess of collisions
> and untracked files?
>

> Just because you don't like some use case doesn't mean it's not gonna
> happen.  Either you prepare for it and make the best of it, or you
> pretend it's not gonna happen and cause extra pain to users.
>

I would argue that I would split the requirements into 3 bands.

1) Must do
2) Should do
3) Nice to have

To me, manually unpacking a tarball and having it recorded in the VDB is a
'nice to have' feature. If we can make it work, great.
I tend to lean with Rich here that recording the data in-band is risky. I
think there is also this premise; that binpkgs can 'maintain VDB
compatibility'. Like I could make a binpkg, wait 2 years, then install it;
and we have to make sure that everything still works.

IMHO its a pretty high cost to pay (tight coupling) for what, to me, is a
nice to have feature.


>
> > > Optimizing for easy recognition
> > > -------------------------------
> > > In order to make it possible for magic-based tools such as file(1) to
> > > easily distinguish Gentoo binary packages from regular tarballs, we
> > > could (ab)use the volume label field, e.g. use:
> > >
> > >   $ tar -V 'gpkg: app-foo/bar-1' -c ...
> > >
> > > This will add a volume label as the first file entry inside the
> tarball,
> > > which does not affect extracting but can be trivially matched via magic
> > > rules.
> > >
> > > Note: this is meant to be used as a method for fast binary package
> > > recognition; I don't think we should reject (hand-modified) binary
> > > packages that lack this label.
> > >
> > >
> > > Optimizing for metadata reading/manipulation performance
> > > --------------------------------------------------------
> > > The main problem with using a single tarball for both metadata and data
> > > is that normally you'd have to decompress everything to reliably unpack
> > > metadata, and recompress everything to update it.  This problem can be
> > > addressed by a few optimization tricks.
> > >
> >
> > These performance goals seem a little bit ill defined.
> >
> > 1) Where are users reporting slowness in binpkg operations?
> > 2) What is the cause of the slowness?
>
> Those are optimizations to not cause slowness compared to the current
> format.  Main use case is recreating package index which would require
> rereading the metadata of all binary packages.


> > Like I could easily see a potential user with many large binpkgs, and the
> > current implementation causing them issues because
> > they have to decompress and seek a bunch to read the metadata out of
> their
> > 1.2GB binpkg. But i'm pretty sure this isn't most users.
> >
> >
> > >
> > > Firstly, all metadata files are packed to the archive before data
> files.
> > >  With a slightly customized unpacker, we can stop decompressing as soon
> > > as we're past metadata and avoid decompressing the whole archive.  This
> > > will also make it possible to read metadata from remote files without
> > > fetching far past the compressed metadata block.
> > >
> >
> > So this seems to basically go against your goals of simple common
> tooling?
>
> No.  My goal is to make it compatible with simple common tooling.  You
> can still use the simple tooling to read/write them.  The optimized
> tools are only needed to efficiently handle special use cases.
>

> > > Secondly, if we're up for some more tricks, we could technically split
> > > the tarball into metadata and data blocks compressed separately.  This
> > > will need a bit of archiver customization but it will make it possible
> > > to decompress the metadata part without even touching compressed data,
> > > and to replace it without recompressing data.
> > >
> > > What's important is that both tricks proposed maintain backwards
> > > compatibility with regular compressed tarballs.  That is, the user will
> > > still be able to extract it with regular archiving tools.
> >
> >
> > So my recollection is that debian uses common format AR files for the
> main
> > deb.
> > Then they have 2 compressed tarballs, one for metadata, and one for data.
> >
> > This format seems to jive with many of your requirements:
> >
> >  - 'ar' can retrieve individual files from the archive.
> >  - The deb file itself is not compressed, but the tarballs inside *are*
> > compressed.
> >  - The metadata and data are compressed separately.
> >  - Anyone can edit this with normal tooling (ar, tar)
> >
> > In short; why should we event a new format?
>
> Because nobody knows how to use 'ar', compared to how almost every
> Gentoo user can use 'tar' immediately?  Of course we could alternatively
> just use a nested tarball but I wanted to keep the possibility
> of actually being able to 'tar -xf' it without having to extract nested
> archives.
>

I think man 'ar' could help them pretty easily. That being said I'm not wed
to 'ar', but trying to show how this problem was solved in a similar domain.


>
> --
> Best regards,
> Michał Górny
>