Re: [gentoo-portage-dev] [RFC] New file layout for PKGDIR and binhosts

[Zac Medico]

On 12/24/2014 10:36 AM, vivo75@×××××.com wrote:
> Il 24/12/2014 17:07, Zac Medico ha scritto:
>> On 12/24/2014 04:01 AM, vivo75@×××××.com wrote:
>>> Il 24/12/2014 09:13, Zac Medico ha scritto:
>>>>> I like this (and it has been a long time coming).  What format are we
>>>>>> going to store the metadata of the use flag combinations and the rest?
>>>> The current approach is to store the data in an xpak segment that is
>>>> appended to the end of the tbz2 file. The $PKGDIR/Packages files serves
>>>> as a cache for the essential parts of the xpak data that are used in
>>>> dependency calculations.
>>>>
>>> I'd like to see the xpak data being put in it's own file at the
>>> _beginning_ of the tar file.
>>>
>>> tar -Jcf \
>>>  ${PKGDIR}/${CATEGORY}/${PN}/${PF}-${COUNTER}.xpak \
>>>  tmp/${CATEGORY}:${PN}:${PF}-${COUNTER}.xpak \
>>>  *all_the_other_stuff*
>>>
>>> this way reading it could be faster on some media and filesystem  and it
>>> would not deviate from the standard tar.
>> There wouldn't be any benefit, because the data is practically always
>> read from the $PKGDIR/Packages cache anyway. The cache is generated when
>> the package is built, and the rate-limiting step there is the building
>> of the package.
> ack, and what about emerge on destination host?

The destination host uses a downloaded copy of $PKGDIR/Packages for
dependency calculations.

Also, I suspect that you're drastically over-estimating the cost of
accessing the xpak metadata, since otherwise you wouldn't be asking this
question.

>>> Being in /tmp/ is only for commodity but the place is debatable.
>>> Instead the fact it _must_ be the first file it's not, in a sequential
>>> archive file like tar some things depend on it.
>> With the current approach, the xpak segment is not part of the tar file.
>> The tar file is compressed, and the xpak segment is appended to the end
>> of the resulting bzip2 file.
>>
>>> seem to be the right time to do the change, since tool need to be
>>> rewritten anyway, but I'll leave to you analyze  the fallout of this change.
>> There will be zero benefits from doing that.
> I see at least two however admittedly not too big
>
> 1) I see having a canonical tarball as an advantage, opposed to be
> forced to use /usr/bin/{qtbz2,qxpak}

You can already use tar to unpack our existing tbz2/xpak files. You only
need special tools if you want to access the package metadata, and I
think it's reasonable to expect anyone who wants to access the metadata
to have the appropriate tools.

> 2) there is a small benefit in space (which increase using xz) for
> bigger packages it would be smaller in percentage.
>
> -rw-r--r-- 1 root root 8720 24 dic 18.25 linuxtv-dvb-headers-5.8.tar.bz2
> -rw-r--r-- 1 root root 9691 24 dic 18.07 linuxtv-dvb-headers-5.8.tbz2

It think this savings is negligible. The environment.bz2 file accounts
for most of the space consumed by the xpak segment, and it is compressed
in order to save space.

> this has been obtained unpacking the xpak _and_ environment.bz2 (to
> avoid double compression) and putting them in a subdirectory
>
> ls -l tmp/linuxtv-dvb-headers-5.8/ usr
> tmp/linuxtv-dvb-headers-5.8/:
> total 80
> -rw-r--r-- 1 root root    11 Dec 24 18:20 BUILD_TIME
> -rw-r--r-- 1 root root     8 Dec 24 18:20 CATEGORY
> -rw-r--r-- 1 root root     2 Dec 24 18:20 DEFINED_PHASES
> -rw-r--r-- 1 root root    52 Dec 24 18:20 DESCRIPTION
> -rw-r--r-- 1 root root     2 Dec 24 18:20 EAPI
> -rw-r--r-- 1 root root   394 Dec 24 18:20 FEATURES
> -rw-r--r-- 1 root root     1 Dec 24 18:20 IUSE
> -rw-r--r-- 1 root root    30 Dec 24 18:20 KEYWORDS
> -rw-r--r-- 1 root root    24 Dec 24 18:20 PF
> -rw-r--r-- 1 root root    31 Dec 24 18:20 RDEPEND
> -rw-r--r-- 1 root root     2 Dec 24 18:20 SIZE
> -rw-r--r-- 1 root root     2 Dec 24 18:20 SLOT
> -rw-r--r-- 1 root root    65 Dec 24 18:20 USE
> -rw-r--r-- 1 root root 22859 Dec 24 18:21 environment
> -rw-r--r-- 1 root root   443 Dec 24 18:20 linuxtv-dvb-headers-5.8.ebuild
> -rw-r--r-- 1 root root     7 Dec 24 18:20 repository

We could save some more space in the xpak segment if we omitted the
ebuild, since environment.bz2 contains everything that we need from the
ebuild. We could also omit FEATURES, since it's non-essential and it's
included in environment.bz2 anyway. However, these are trivial
micro-optimizations.
-- 
Thanks,
Zac