[gentoo-portage-dev] Re: [RFC] Improving Gentoo package format

[Duncan]

Francesco Riosa posted on Sun, 11 Nov 2018 17:05:37 +0100 as excerpted:

> Il giorno dom 11 nov 2018 alle ore 09:29 Michał Górny
> <mgorny@g.o>
> ha scritto:
>
>> On Sat, 2018-11-10 at 09:37 -0500, Alec Warner wrote:
>>> On Sat, Nov 10, 2018 at 8:09 AM Michał Górny <mgorny@g.o>
>>> wrote:
>> [...]
>>>> My proposal ===========
>>>>
>>>> Basic format ------------
>>>> The base of the format is a regular compressed tarball.
>>>> There's no junk appended to it but the metadata is stored
>>>> inside it as /var/db/pkg/${PF}.  The contents are as compatible
>>>> with the actual vdb format as possible.
>>>>
>>>>
>>> Just to clarify, you are suggesting we store the metadata inside
>>> the contents of the binary package itself (e.g. where the other
>>> files that get merged to the liveFS are?) What about collisions?
>>>
>>> E.g. I install 'machine-images/gentoo-disk-image-1.2.3' on a machine
>>> that already has 'machine-images/gentoo-disk-image-1.2.3' installed,
>>> won't it overwrite files in the VDB at qmerge time?
>>
>> Portage will obviously move the files out, and process them as
>> metadata.

>> The idea is to precisely use a directory that can't be normally part
>> of binary packages, so can't cause collisions with real files (even if
>> they're very unlikely to ever happen).
>>
>>>> This has the following advantages:
>>>>
>>>> + Binary package is still stored as a single file.

Breaking these down into RFC style MUST/SHOULD/MAY levels (as already 
suggested elsewhere), for me, this is...

SHOULD/MAY

(Would be a MAY, nice to have, but the existing solution has it, thus 
arguably raising the priority to SHOULD.)

>>>> + It uses a standard compressed .tar format, with minimal
>>>> customization.

MUST

(Losing the existing functionality here would be horrible.  FWIW I 
routinely use binpkgs as a reference, for "clean" config files, comparing 
install trees of old and new versions, etc.  Having tools that allow 
browsing standard compressed tar archives as virtual extensions to the 
filesystem makes that a breeze. =:^)

>>>> + The user can easily inspect and modify the packages with standard
>>>> tools (tar and the compressor).

MUST

(As pointed out, portage itself already does this when doing binpkg 
moves, etc.  Losing that would be horrible!)

>>>> + If we can maintain reasonable level of vdb compatibility, the
>>>> user can even emergency-install a package without causing too much
>>>> hassle (as it will be recorded in vdb); ideally Portage would
>>>> detect this vdb entry and support fixing the install afterwards.
>>>>
>>>>
>>> I'm not certain this is really desired.

SHOULD/MAY

(I'd say SHOULD, but while possible to emergency-install via untarring 
now, portage doesn't do anything with it at all, so the detect and fix 
functionality is a bonus, thus arguably lowering it to a MAY.)

>> Are you saying it's better that user emergency-installs a package
>> without recording it in vdb, and ends up with mess of collisions and
>> untracked files?
>>
>> Just because you don't like some use case doesn't mean it's not gonna
>> happen.  Either you prepare for it and make the best of it, or you
>> pretend it's not gonna happen and cause extra pain to users.

I think I've had to do this twice in ~1.5 decades, plus once reaching 
into the tarball to extract a single file that was broken in a newly 
installed glibc, breaking portage (and much of the system, but bunzip 
still worked!) so I couldn't undo it using portage.

The first time I didn't know enough to clean up manually, but the second 
time (and the reach-in time) I did.  It'd *definitely* be nice to have 
portage be able to clean up automatically.

> Another option would be to install in a near but not overlapping
> directory,
> example:
> /var/db/pkg/${PF}-binpkg
>
> this way the user that know what to do with that data can play with it,
> also portage could be instructed to stat() that directory and take
> action (halt maybe?) if present.

Idea ++

Detect and fix has already been proposed, but detect and halt with an 
error and a pointer to manual fix instructions is arguably already better 
than current.

Which suggests an easy implementation split, delaying the "fix" step 
until later, if it would complicate the initial implementation too much.

[Bikeshed]  I was thinking binpkg-${PF} to emphasize the binpkg part and 
group any emergency-installed packages together in an alphabetical 
listing.  But whichever's easiest for portage to work with, which 
probably makes the -binpkg suffix version a better choice, requiring less 
modification to existing code.


Is there any interest at all in binpkgs, perhaps when improved, from the 
other PMs?  Or are they effectively dead now or not interested in binpkgs 
even if the format were to be improved, or simply too hard to work with?  
Because "it'd be nice" (aka MAY level) to have this formally standardized 
to PMS... if there's any interest from the other PMs.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman