Gentoo Archives: gentoo-portage-dev

From: Brian Harring <ferringb@×××××.com>
To: gentoo-portage-dev@l.g.o
Subject: Re: [gentoo-portage-dev] Package compression header for binhosts
Date: Tue, 01 Jun 2010 21:52:28
In Reply to: Re: [gentoo-portage-dev] Package compression header for binhosts by Zac Medico
1 On Tue, Jun 1, 2010 at 2:37 PM, Zac Medico <zmedico@g.o> wrote:
3 > On 06/01/2010 02:22 PM, Brian Harring wrote:
4 > > As for zacs tool to try and generate new views of a repository via
5 > > hardlinking/recreating the tree... frankly it's a bit of a hack. Via
6 > > DEFAULT_URI and relying on the hash, you can make a stable repository
7 > that
8 > > is able to be updated in place without corrupting ongoing downloads-
9 > simply
10 > > put, new additions to the repo don't perturb current DL's since the md5
11 > is
12 > > the same (hash collision chance is low enough that I don't care about it
13 > > here).
14 >
15 > When you say "hash collision" are you talking about
16 > Maybe that behavior is acceptable for
17 > small-scale private use, but for large scale public repositories I'd
18 > say it's totally unacceptable.
21 That bug isn't about a collision, it's about files being replaced underneath
22 Packages feet. Even with the tricks you've leveled the issue of things
23 changing under foot still is possible- you've just made the race less
24 likely.
26 What I was talking about was solving this issue once and for all via
27 restructuring, and specifically refering to the potential of an md5
28 collision in the URI space- specifically what I'm implementing for pkgcore
29 is the ability to do stupid stuff like this-
31 http://host/binpkg-store/$MD5.{txz,tbz2,tgz}
33 then have multiple views accessible just via pointing the binpkg repo remote
34 url at
36 http://host/views/license/oss-approved/
37 http://host/views/keywords/amd64/stable/
38 http://host/views/raw/ # no filtering on the view of the binpkg repo, see
39 everything.
41 Via restructuring where the binpkgs are stored and doing this approach,
42 multiple views can be had easily into the repo. An additional benefit of
43 this approach is that via making URI able to point outside the host, you
44 could combine multiple seperate repositories into one just via a view.
48 > Eventually, I'd like to see gentoo
49 > officially distributing binary packages, so that we'll be able to
50 > get a slice of the binary distribution pie. When that happens, we're
51 > certainly not going to want to have race conditions like these in
52 > our public binhosts.
53 >
55 I'd suggest abandoning the current repository layout of Packages then, since
56 it's irrevocably flawed. You can hack around it via jamming timestamp/md5
57 info into URI, but that's not a sane solution.
59 ~harring