Gentoo Archives: gentoo-dev

From: Alexey Sokolov <alexey+gentoo@××××××××.org>
To: gentoo-dev@l.g.o
Cc: binhost@g.o
Subject: Re: [gentoo-dev] New project: binhost
Date: Wed, 10 Feb 2021 23:20:12
Message-Id: CAHK_jdhz7GcQ+1zTz4pPJ7EJ72Qk0CPbgyTsk=pvdL8fYR9X4w@mail.gmail.com
In Reply to: Re: [gentoo-dev] New project: binhost by Rich Freeman
1 ср, 10 февр. 2021 г. в 19:12, Rich Freeman <rich0@g.o>:
2 >
3 > On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel <dilfridge@g.o> wrote:
4 > >
5 > > * what portage features are still needed or need improvements (e.g. binpkg
6 > > signing and verification)
7 > > * how should hosting look like
8 >
9 > Some ideas for portage enhancements:
10 >
11 > 1. Ability to fetch binary packages from some kind of repo.
12 > 2. Ability to have multiple binary packages co-exist in a repo (local
13 > or remote) with different build attributes (arch, USE, CFLAGS,
14 > DEPENDS, whatever).
15 > 3. Ability to pick the most appropriate binary packages to use based
16 > on user preferences (with a mix of hard and soft preferences).
17 >
18 > One idea I've had around how #2-3 might be implemented is:
19 > 1. Binary packages already contain data on how they were built (USE
20 > flags, dependencies, etc). Place this in a file using a deterministic
21 > sorting/etc order so that two builds with the same settings will have
22 > the same results.
23 > 2. Generate a hash of the file contents - this can go in the filename
24 > so that the file can co-exist with other files, and be located
25 > assuming you have a full matching set of metadata.
26 > 3. Start dropping attributes from the file based on a list of
27 > priorities and generate additional hashes. Create symlinked files to
28 > the original file using these hashes (overwriting or not existing
29 > symlinks based on policy). This allows the binary package to be found
30 > using either an exact set of attributes or a subset of higher-priority
31 > attributes. This is analogous to shared object symlinking.
32 > 4. The package manager will look for a binary package first using the
33 > user's full config, and then by dropping optional elements of the
34 > config (so maybe it does the search without CFLAGs, then without USE
35 > flags). Eventually it aborts based on user prefs (maybe the user only
36 > wants an exact match, or is willing to accept alternate CFLAGs but not
37 > USE flags, or maybe anything for the arch is selected.
38
39 A related idea: if user could specify which USE-flags are mandatory to
40 be set, which USE flags are mandatory to be not set, and which can be
41 either way, it's easier to find the matching binary package with less
42 constraints, where only some flags match.
43
44 > 5. As always the final selected binary package still gets evaluated
45 > like any other binary package to ensure it is usable.
46 >
47 > Such a system can identify whether a potentially usable file exists
48 > using only filename, cutting down on fetching. In the interests of
49 > avoiding useless fetches we would only carry step 3 reasonably far -
50 > packages would have to match based on architecture and any dynamic
51 > linking requirements. So we wouldn't generate hashes that didn't
52 > include at least those minimums, and the package manager wouldn't
53 > search for them.
54 >
55 > Obviously you could do more (if you have 5 combinations of use flags,
56 > look for the set that matches most closely). That couldn't be done
57 > using hashes alone in an efficient way. You could have a small
58 > manifest file alongside the binary package that could be fetched
59 > separately if the package manager wants to narrow things down and
60 > fetch a few of those to narrow it down further.
61 >
62 > Or you could skip the hash searching and just fetch all the manifests
63 > for a particular package/arch and just search all of those, but that
64 > is more data to transfer just to do a query. A metadata cache of some
65 > kind of might be another solution. Content hashes would probably
66 > still be useful just to allow co-existence of alternate builds.
67 >
68 > --
69 > Rich
70 >