Re: [RFC] gpkg format proposal v2 (was: Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format)

[Michał Górny]

On Sun, 2018-11-11 at 21:53 +0100, Michał Górny wrote:
> Hi,
>
> Ok, here's the second version integrating the feedback received.
> The format is much simpler, based on nested tarballs inspired by Debian.
>
> The outer tarball is uncompressed and uses '.gpkg.tar' suffix.  It
> contains (preferably in order but PM should also handle packages with
> mismatched order):
>
> 1. Optional (but recommended) "gpkg: ${PF}" package label that can be
> used to quickly distinguish Gentoo binpkgs from regular tarballs
> (for file(1)).
>
> 2. "metadata.tar${comp}" tarball containing binary package metadata
> as files.
>
> 3. Optional "metadata.tar${comp}.sig" containing detached signature
> for the metadata archive.
>
> 4. "contents.tar${comp}" tarball containing files to be installed.
>
> 5. Optional "contents.tar${comp}.sig" containing detached signature for
> the contents archive.
>
> Notes:
>
> a. ${comp} can be any compression format supported by binary packages.
> Technically, metadata and content archives may use different
> compression.  Either or both may be uncompressed as well.
>
> b. While signatures are optional, the PM should have a switch
> controlling whether to expect them, and fail hard if they're not present
> when expected.
>
>
> Advantages
> ----------
> Guaranteed:
>
> + The binary package is still one file, so can be fetched easily.
>
> + File format is trivial and can be extracted using tar(1) + compressor.
>
> + The metadata and contents are compressed independently, and so can be
> easily extracted or modified independently.
>
> + The package format provides for separate metadata and content
> signatures, so they can be verified independently.
>
> + Metadata can be compressed now.
>
> Achieved by regular archives (but might be broken if modified by user):
>
> + Easy recognition by magic(1).
>
> + The metadata archive (and its signature) is packed first, so it may be
> read without fetching the whole binpkg.
>
>
> Why not .ar format?
> -------------------
> The use of .ar format has been proposed, akin to Debian.  While
> the option is mostly feasible, and the simplicity of .ar format would
> reduce the outer size of binary packages, I think the format is simply
> too obscure.  It lives mostly as static library format, and the tooling
> for it is part of binutils.  LSB considers it deprecated.  While I don't
> see it going away anytime soon, I'd rather not rely on it in order to
> save a few KiB.
>
>
> Is there anything left to address?
>

Hmm, I've missed one disadvantage compared to xpak and v1: at least with
the standard tools, we can't build the binary package on the fly without
creating temporary archives (and therefore duplicating disk space use).

In other words, xpak and v1 formats made it possible to tar
the installation image straight to the new package.

The v2 format requires creating "contents.tar${comp}" first, and then
creating the actual binary package with it.  I don't think we can avoid
this without creating a custom .tar writing tool that supports adding
data on-the-fly (e.g. by writing the file data, then seeking back to
update the size record).

Of course, one option would be to use ZIP ;-).

-- 
Best regards,
Michał Górny