Gentoo Archives: gentoo-portage-dev

From: Zac Medico <zmedico@g.o>
To: gentoo-portage-dev@l.g.o
Subject: Re: [gentoo-portage-dev] Package compression header for binhosts
Date: Tue, 01 Jun 2010 23:54:13
In Reply to: Re: [gentoo-portage-dev] Package compression header for binhosts by Brian Harring
1 On 06/01/2010 02:52 PM, Brian Harring wrote:
2 > That bug isn't about a collision, it's about files being replaced underneath
3 > Packages feet. Even with the tricks you've leveled the issue of things
4 > changing under foot still is possible- you've just made the race less
5 > likely.
7 AFAIK the race is completely eliminated by the RCU-like snapshot
8 mechanism. I think I like your hash-in-the-filename idea better
9 though, since it seems simpler to implement and maintain. It can
10 even be done with the existing version 0 format by abusing the
11 per-package PATH attribute to refer to a filename that contains a
12 hash (maybe something like $CATEGORY/$PF.$MD5.tbz2). I wouldn't want
13 to abuse PATH for compression in version 0 though, since clients are
14 only required to support tbz2.
16 > What I was talking about was solving this issue once and for all via
17 > restructuring, and specifically refering to the potential of an md5
18 > collision in the URI space- specifically what I'm implementing for pkgcore
19 > is the ability to do stupid stuff like this-
20 >
21 > http://host/binpkg-store/$MD5.{txz,tbz2,tgz}
23 That would be the MD5 of the entire file, after compression and
24 having the xpak segment appended, right?
26 > then have multiple views accessible just via pointing the binpkg repo remote
27 > url at
28 >
29 > http://host/views/license/oss-approved/
30 > http://host/views/keywords/amd64/stable/
31 > http://host/views/raw/ # no filtering on the view of the binpkg repo, see
32 > everything.
34 So, the default path of a package would come from looking at the MD5
35 in the Packages file and then mapping that to a path?
37 > Via restructuring where the binpkgs are stored and doing this approach,
38 > multiple views can be had easily into the repo. An additional benefit of
39 > this approach is that via making URI able to point outside the host, you
40 > could combine multiple seperate repositories into one just via a view.
42 This might also be useful for creating per-profile views while
43 allowing packages to be shared between profiles in cases when
44 hosting a separate build would be redundant. It might be possible to
45 save lots of build time, disk space, and testing that way.
47 Being able to have multiple builds of the same package with
48 different USE settings is also solves bug 150031 [1].
50 >> Eventually, I'd like to see gentoo
51 >> officially distributing binary packages, so that we'll be able to
52 >> get a slice of the binary distribution pie. When that happens, we're
53 >> certainly not going to want to have race conditions like these in
54 >> our public binhosts.
55 >>
56 >
57 > I'd suggest abandoning the current repository layout of Packages then, since
58 > it's irrevocably flawed. You can hack around it via jamming timestamp/md5
59 > info into URI, but that's not a sane solution.
61 Shrug, it's a handy way to solve race conditions given the existing
62 version 0 format. It's not optimal, so we'll surely want something
63 better in version 1.
65 [1]
66 --
67 Thanks,
68 Zac


Subject Author
Re: [gentoo-portage-dev] Package compression header for binhosts Brian Harring <ferringb@×××××.com>