Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/20/21 8:17 PM, Zac Medico wrote:
> On 2/13/21 4:53 PM, Zac Medico wrote:
>> On 2/13/21 4:37 PM, Zac Medico wrote:
>>> On 2/11/21 1:17 AM, Michał Górny wrote:
>>>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'm announcing a new project here - "binhost"
>>>>>>
>>>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>>>> precompiled packages for a subset of configurations, via central
>>>>>> binary package hosting. Currently we are still in the conceptual
>>>>>> planning stage. "
>>>>>>
>>>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>>>
>>>>>> If you're interested in helping out, feel free to add yourself on the
>>>>>> wiki page.
>>>>>>
>>>>>> Note that I see actually *building* the packages not as the central
>>>>>> point of the project (that could be e.g. a side effect of a
>>>>>> tinderbox). I'm more concerned about
>>>>>> * what configurations should we use
>>>>>> * what portage features are still needed or need improvements (e.g.
>>>>>> binpkg signing and verification)
>>>>>> * how should hosting look like
>>>>>> * and how we can test this on a limited scale before it goes "into
>>>>>> production"
>>>>>> * ...
>>>>>>
>>>>>> Comments, ideas, flamebaits? :D
>>>>>>
>>>>>> Cheers,
>>>>>> Andreas
>>>>>>
>>>>>
>>>>> It would be great to improve portage speed with handling binpkgs. I
>>>>> already have my own binhost for a couple of Gentoo systems and even
>>>>> though these systems don't have to compile anything themselves,
>>>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>>>> installation time. Arch Linux' pacman only takes a fraction of this
>>>>> time for the very same task.
>>>>> I know that I compare apples with pears here but even reducing the
>>>>> current portage time by 50% would be a huge improvement.
>>>>
>>>> Is that really a problem?  For me, Portage takes about an hour just to
>>>> do the dependency processing these days.  In fact, building from sources
>>>> is now faster than dependency calculations.
>>>
>>> The ratio of these times is dependent on the complexity of the
>>> dependencies involved, and so is the answer to your question.
>>
>> Also, in the context of binary packages, dependencies calculations are
>> generally simpler for binary packages because their USE conditionals and
>> slot-operator := dependencies are frozen in a particular state. This
>> dramatically reduces the search space involved in dependency calculation.
>
> IUSE_RUNTIME will obviously introduce conditionals in binary package
> dependencies, but we should welcome these conditionals because they will
> provide useful flexibility.
>
> I think IUSE_RUNTIME will be a very nice feature to have for project
> binhost, since it will allow binary package dependencies to behave with
> flexibility that more closely resembles the flexibility of ebuild
> dependencies.

We can borrow paludis's notion of pbins [1] to generate ebuilds which
install pre-built content, and the generated ebuilds could have USE
flags that behave similarly to IUSE_RUNTIME in the sense that changes to
USE flags will result in different builds of pre-built content being
installed. A content-hash distfiles layout [2] could serve as a
convenient way to store separate builds of pre-built content for
multiple combinations of USE flags, and a generated ebuild would index
the build by USE flag combination.

Also, for the generated ebuilds, we can generate USE flags to include
separate SRC_URI downloads for pre-built content to support things like
FEATURES=split-debug and FEATURES=install-sources.

[1] https://paludis.exherbo.org/overview/pbins.html
[2] https://bugs.gentoo.org/756778
-- 
Thanks,
Zac