1 |
On 12/23/2014 07:51 PM, Zac Medico wrote: |
2 |
> Hi, |
3 |
> |
4 |
> As discussed in bug 150031 [1], it would be useful if PKGDIR could |
5 |
> accommodate multiple binary packages built from the same source ebuild. |
6 |
> Use cases for preserving multiple builds typically involve supporting |
7 |
> multiple clients (with partially compatible configurations) from a |
8 |
> single unified binhost. In this context, some of the reasons to retain |
9 |
> multiple builds are: |
10 |
> |
11 |
> * Different USE flag combinations enabled (--newuse/--binpkg-respect-use |
12 |
> needed) |
13 |
> |
14 |
> * Different versions of installed dependencies (EAPI 5 slot := operators |
15 |
> needed) |
16 |
> |
17 |
> * Different repositories/overlays, with variance in the time of the last |
18 |
> sync (--changed-deps/--binpkg-changed-deps needed if dependencies change |
19 |
> due to eclass changes or ebuild modifications without revbump) |
20 |
> |
21 |
> Given the above variety of reasons to retain previous builds, a simple |
22 |
> counter (1, 2, 3,...) seems like a reasonable means to generate unique |
23 |
> file names. |
24 |
> |
25 |
> In order to avoid having too many files in a directory, we can use a |
26 |
> separate directory for each ${CATEGORY}/${PN}, like we do for the source |
27 |
> ebuild repositories. |
28 |
> |
29 |
> In order to avoid having to deal with multiple file extensions for |
30 |
> different compression types, we can simply use .xpak for the file |
31 |
> extension [2], since that's the name of the format that we use to append |
32 |
> metadata to our existing tbz2 files. We can simply probe the first few |
33 |
> bytes of the file in order to determine the compression type: |
34 |
> |
35 |
> gzip: 1f 8b |
36 |
> bzip2: 42 5a 68 39 |
37 |
> xz: fd 37 7a 58 5a 00 |
38 |
> |
39 |
> Users will be able change their compression settings at any time, but |
40 |
> the .xpak file extension will remain constant regardless of that |
41 |
> setting. It won't matter if they have a mixture of files compressed with |
42 |
> different compressors. |
43 |
> |
44 |
> A tool like eclean-pkg will be needed to clean up old binary packages |
45 |
> based on user preferences. We might also provide a variety of on-the-fly |
46 |
> garbage collection settings. |
47 |
> |
48 |
> Based on the above discussion, the location of any particular binary |
49 |
> package can be expressed as follows: |
50 |
> |
51 |
> ${PKGDIR}/${CATEGORY}/${PN}/${PF}-${COUNTER}.xpak |
52 |
> |
53 |
> The existing format of the ${PKGDIR}/Packages index will work fine, |
54 |
> since it allows each package to specify a PATH attribute which |
55 |
> corresponds to the path of the file relative to the base directory. If |
56 |
> the .xpak files use bzip2 compression, it will even be compatible with |
57 |
> existing clients (though they won't be able to intelligently choose |
58 |
> between multiple packages of the same version). If all the packages of a |
59 |
> given version are ordered by ${COUNTER}, then existing clients will |
60 |
> simply download the latest build. |
61 |
> |
62 |
> [1] https://bugs.gentoo.org/show_bug.cgi?id=150031 |
63 |
> [2] http://dev.gentoo.org/~zmedico/portage/doc/man/xpak.5.html |
64 |
> |
65 |
I like this (and it has been a long time coming). What format are we |
66 |
going to store the metadata of the use flag combinations and the rest? |
67 |
I guess that's already stored since portage knows not to use binpkgs if |
68 |
those change. |
69 |
|
70 |
Also, would this change be a good time to change to store that metadata |
71 |
externally? Running portage over NFS with binpkgs takes forever, I |
72 |
don't think a binhost makes it faster either. If there were some way to |
73 |
get all the info for the binpkgs into one file (so it could be run on |
74 |
cron or something), this could mean that I'd only have to do one file |
75 |
request for all that metadata and would be much quicker than inspecting |
76 |
all those files. |
77 |
|
78 |
-- |
79 |
-- Matthew Thode (prometheanfire) |