Re: [gentoo-dev] [RFC] New ebuild metadata variables for filtering of binary package soname dependencies (analogous to Fedora __provides_exclude and __requires_exclude)

[Zac Medico]

On 12/29/2014 03:28 AM, Matthias Maier wrote:
>
> Am 29. Dec 2014, 09:46 schrieb Zac Medico <zmedico@g.o>:
>
>> In order to implement something like this for Gentoo, we could add
>> PROVIDES_EXCLUDE and REQUIRES_EXCLUDE ebuild metadata variables. These
>> variables would be used to automatically generate PROVIDES and REQUIRES
>> data for binary packages, based on the output of scanelf. The resulting
>> PROVIDES and REQUIRES data would simply contain lists of sonames (and
>> their corresponding ABIs, for multilib support), which would be used to
>> resolve dependencies of binary packages. For example, the resulting
>> REQUIRES and PROVIDES data for sys-libs/zlib-1.2.8-r1 built with
>> ABI_X86="32 64" might look something like this:
>>
>> 	REQUIRES: X86_64: libc.so.6 386: libc.so.6
>>
>> 	PROVIDES: X86_64: libz.so.1 386: libz.so.1
>
> So this basically shifts information that is available only after a
> merge (and  e.g. checked in the preserved-libs feature) to the metadata
> of the binary package?

Right.

>> For libz, the new PROVIDES_EXCLUDE and REQUIRES_EXCLUDE ebuild variables
>> would not be needed. However, for things like python and perl modules,
>> these variables would be used to exclude unwanted sonames from REQUIRES
>> and PROVIDES.
>
> How do you intend to use the new metadata in detail?

The REQUIRES and PROVIDES data will supplement *DEPEND when resolving
dependencies for binary packages.

There is some overlap in information between EAPI 5 slot-operator
dependencies and REQUIRES/PROVIDES. That's a good thing though, for at
least the following 3 reasons:

1) Slot-operator dependencies have not reached 100% adoption, so
REQUIRES/PROVIDES soname dependencies will provide information for all
packages, regardless of EAPI 5 slot-operator and sub-slot adoption. We
can enable PROVIDES_EXCLUDE and REQUIRES_EXCLUDE support for all
existing EAPIs, so that migrating packages to a new EAPI will not be a
barrier to adoption of PROVIDES_EXCLUDE and REQUIRES_EXCLUDE.

2) Gentoo has a long-standing policy of omitting system packages from
dependencies. This leads to unspecified dependencies that might never be
specified with slot-operators and sub-slots. For these unspecified
dependencies, REQUIRES/PROVIDES soname dependencies will provide
dependency information that is not currently available by any other means.

3) REQUIRES/PROVIDES soname dependencies can be used to implement QA
checks for dependencies that have been accidentally omitted from *DEPEND
variables.

> Is it planned to introduce this as another dependency constraint on
> binary merges? Will this become a 'hard' or 'soft' dependency?

It should be considered a 'hard' dependency. Of course, we could provide
an option to make it 'soft', but I do not anticipate a need for this.

I think we should add something like the
--ignore-built-slot-operator-deps option which will allow the user to
explicitly disable rebuilds triggered by soname changes. We could even
re-purpose --ignore-built-slot-operator-deps so that it also applies to
rebuilds triggered by REQUIRES/PROVIDES data, but it's probably better
to use a separate option.

> E.g. consider two packages, libfoo-1 (soname libfoo-1) and bar linking
> against libfoo. And libfoo-2 (with new soname libfoo-2) emerged
> replacing version 1 on the binhost replacing libfoo-1, bar not rebuild.
> We now have on the binhost:
>   libfoo-1 (soname libfoo-1)
>   libfoo-2 (soname libfoo-2)
>   bar (dynamically linked to libfoo-1)
>
>  (i)   Currently, an update on a client (using the binhost) will upgrade
>        libfoo-1 to libfoo-2 and the preserved-libs feature will preserve
>        the necessary shared library libfoo-1 for bar.

This behavior will remain the same as long as FEATURES=preserve-libs is
enabled on the client.

>  (ii)  What should happen with above metadata?

Any libraries preserved by preserve-libs will have no effect on the
REQUIRES/PROVIDES data. So, the metadata will be exactly the same,
regardless of preserve-libs settings on either the client or server. You
can consider preserve-libs as being beyond the scope of the whole
proposal, and essentially irrelevant to it.

> Shall an update only resolve to libfoo-1 and bar?

It depends on the state of the client and its configuration settings.
Under the most common usage scenario, the client should be configured to
skip the libfoo-2 update until the binhost has rebuilt bar to link
against libfoo-2. Of course, the client would also have the option to
pull in the libfoo-1 update and rebuild bar itself. If bar uses a
slot-operator dependency and libfoo uses sub-slots, the usual
slot-operator mechanism would trigger rebuild on the client. If libfoo
has not adopted sub-slots, then the PROVIDES data of libfoo-2 and the
REQUIRES data of the installed bar instance would serve as an
alternative mechanism to trigger rebuild on the client.

>  (iii) What if libfoo-1 is removed from the binhost? If libfoo-1 is
>        already merged on the client an update wrt (i) is still possible.
>        So, shall the preserved-libs feature be used as a fall back then?

The client should try to eliminate any preserved-libs at the earliest
opportunity. Therefore the most common usage scenario would be
essentially the same as described above with respect to (i).

>  (iv)  If bar is rebuild on the binhost - which it should be nowadays -
>        above metadata information could be used to automatically remerge
>        bar on the client - but couldn't this already be done with a
>        slot-operator dependency?

Yes, but we should also consider the 3 reasons that I have enumerated
above, which make REQUIRES/PROVIDES data extremely useful for plenty of
cases that are not currently covered by slot-operator dependencies (and
will not be covered for the foreseeable future).

> Further, we currently have multiple approaches for ensuring consistency
> of dynamic linkage, the "poor man's approach" revdep-rebuild, language
> specific update scripts (for python, perl, haskell,...), the
> preserved-libs feature and, finally, slot-operators.
> This would be another iteration  of a solution, this time for binary
> merges. Can you give me an example where your proposed approach is the
> only way to ensure consistency of dynamic linkage, or is conceptually
> cleaner?

Conceptually, slot-operators are the cleanest solution. The
preserve-libs feature complements slot-operators by temporarily
satisfying soname dependencies during the process of updating packages.

However, due to the 3 reasons that I have enumerated above,
REQUIRES/PROVIDES data is extremely useful for plenty of cases that are
not currently covered by slot-operator dependencies (and will not be
covered for the foreseeable future).

> I, for my part, like the fact that such metadata generation would be
> completely automatic.

Yes, I think that this proposal represents a very practical way to
integrate binary soname dependencies with Gentoo, so that binary package
support will be more competitive with binary distros.
-- 
Thanks,
Zac