Re: [gentoo-portage-dev] [PATCH] Generate soname dependency metadata (282639)

[Zac Medico]

On 01/29/2015 04:54 PM, Anthony G. Basile wrote:
> On 01/26/15 22:16, Zac Medico wrote:
> ELF gets us like 99% of the way but will eventually have to think about
> Mach-O for gentoo-on-mac etc.  Maybe COFF, a.out.  Not sure where to
> draw the line.

Yeah, I guess we can let the Prefix team decide on that stuff.

> I'm a bit confused here.  So PROVIDES and REQUIRES are new files in
> /var/db/pkg/<cat>/<pkg>?

Right.

> And the format of NEEDED.ELF.2 is not changing?  Correct?

It has one new field, for the multilib category (same as the abstract
abi identifier discussed in bug 534206). I didn't bother to document it,
since is was already completely undocumented. I suppose it would be nice
to have it documented in portage.5, though.

> We should note at some point that arm_32 means EABI and not OABI which
> are two different 32-bit arm abis.  We don't support oabi in gentoo
> anymore, but still this can be confusing.  If it did come up in some
> context, we could distinguish it as arm_o32 like mips below.

Yeah, I'm inclined to just wait for someone to request support for
something like this, and deal with it then.

> We probably don't have to note that arm_64 is aarch6 which is really a
> different ISA, not backwards compatible to 32-bit arm.  As you say below
> this is just a naming convention, but in every other case each line does
> refer to one ISA, except maybe ia_32 can be confused with x86.

Well, wikipedia says "AArch64 provides user-space compatibility with
ARMv7-A ISA" [1], so I think that's a good enough reason to stick with
arm_{32,64} for now.

>> +#    hppa_{32,64}
>> +#    ia_{32,64}
>> +#    m68k_{32,64}
>> +#    mips_{eabi32,eabi64,n32,n64,o32,o64}
>> +#    ppc_{32,64}
>> +#    s390_{32,64}
>> +#    sh_{32,64}
>> +#    sparc_{32,64}
>> +#    x86_{32,64,x32}
>> +#
>> +# NOTES:
>> +#
>> +# * The ABIs referenced by some of the above *_32 and *_64 categories
>> +#   may be imaginary, but they are listed anyway, since the goal is to
>> +#   establish a naming convention that is as consistent and uniform as
>> +#   possible.
>> +#
>> +# * The Elf header's e_ident[EI_OSABI] byte is completely ignored,
>> +#   since OS-independence is one of the goals. The assumption is that,
>> +#   for given installation, we are only interested in tracking multilib
>> +#   ABIs for a single OS.
>
> If you run readelf -h on (say) bash in any of our stage3's tarballs you
> always get "OS/ABI: UNIX - System V" irrespective of arch and abi.  I
> don't know what you would get on BSD, but the field is totally
> irrelevant for our purposes despite the name.  As far as I can tell, it
> is totally invariant across arches and abis.
>
> You can even unpack the the stage3's on an amd64 host and run readelf
> form the host on the chroot target and you'll get the elf header, so you
> don't need access to native hardware.
>
> The comment suggests that there might be some interesting information
> there, but there isn't.  Maybe I'm just reading too much into it.

Well, a quick google search seems to indicate that FreeBSD uses
EI_OSABI. I was specifically thinking about FreeBSD when I wrote that
comment, because I was aware that Gentoo/FBSD was using ELF, and I just
assumed that they would have a different EI_OSABI than Linux.

>> diff --git a/pym/portage/util/_dyn_libs/NeededEntry.py
>> b/pym/portage/util/_dyn_libs/NeededEntry.py
>> new file mode 100644
>> index 0000000..5de59a0
>> --- /dev/null
>> +++ b/pym/portage/util/_dyn_libs/NeededEntry.py
>> @@ -0,0 +1,83 @@
>> +# Copyright 2015 Gentoo Foundation
>> +# Distributed under the terms of the GNU General Public License v2
>> +
>> +from __future__ import unicode_literals
>> +
>> +import sys
>> +
>> +from portage import _encodings, _unicode_encode
>> +from portage.exception import InvalidData
>> +from portage.localization import _
>> +
>> +class NeededEntry(object):
>> +    """
>> +    Represents one entry (line) from a NEEDED.ELF.2 file. The entry
>> +    must have 5 or more semicolon-delimited fields in order to be
>> +    considered valid. The sixth field is optional, corresponding
>> +    to the multilib category. The multilib_category attribute is
>> +    None if the corresponding field is either empty or missing.
>> +    """
>> +
>> +    __slots__ = ("arch", "filename", "multilib_category", "needed",
>> +        "runpaths", "soname")
>
> Looks like this answers my question above about the format of NEEDED.ELF.2

Yeah, maybe this is enough documentation for those who would be interested.

> Nice logic here :)  The only thing that I don't get is why we might need
> {provides,requires}_exclude patterns.  I guess its good design
> principles but I can't think of a use case.

I copied the idea from Fedora [2]. Since the soname dependencies are
automatically generated, we want to be able to filter them if necessary,
though hopefully it won't be needed very often.

>> b/pym/portage/util/elf/constants.py
>> new file mode 100644
>> index 0000000..3857b71
>> --- /dev/null
>> +++ b/pym/portage/util/elf/constants.py
>
> Document where these are coming from else we'll loose the connection to
> the standard.  They should all be in <elf.h> provided by elfutils, but
> I'm sure standard is documented somewhere officially.  Even if you just
> say "see <elf.h>" that might be enough to clue people where to look for
> these definitions.

Okay, will do. I wasn't really sure where they originated, since so many
packages seem to have an elf.h, but elfutils sounds good to me.

>> diff --git a/pym/portage/util/elf/header.py
>> b/pym/portage/util/elf/header.py
>> new file mode 100644
>> index 0000000..3310eeb
>> --- /dev/null
>> +++ b/pym/portage/util/elf/header.py
>> @@ -0,0 +1,62 @@
>
> Looks good.  I'm going to perf test this but I don't think it will be
> too big a hit.  I don't know how we would get to this point in the code
> tree, but let me ask you this: am I right in thinking you won't hit
> ELFHeader.read() with every file that's being installed by portage?
> You'll only get here for elf objects?  Correct?

Yeah, it's only called in doebuild.py for the ELF files that were
previously recorded in NEEDED.ELF.2, based on scanelf output.

[1] http://en.wikipedia.org/wiki/ARM_architecture#ARMv8-A
[2] http://fedoraproject.org/wiki/Packaging:AutoProvidesAndRequiresFiltering
-- 
Thanks,
Zac