Re: [gentoo-portage-dev] Package compression header for binhosts

[Brian Harring]

On Tue, Jun 1, 2010 at 2:37 PM, Zac Medico <zmedico@g.o> wrote:

> On 06/01/2010 02:22 PM, Brian Harring wrote:
> > As for zacs tool to try and generate new views of a repository via
> > hardlinking/recreating the tree... frankly it's a bit of a hack.  Via
> > DEFAULT_URI and relying on the hash, you can make a stable repository
> that
> > is able to be updated in place without corrupting ongoing downloads-
> simply
> > put, new additions to the repo don't perturb current DL's since the md5
> is
> > the same (hash collision chance is low enough that I don't care about it
> > here).
>
> When you say "hash collision" are you talking about
> http://crosbug.com/3225? Maybe that behavior is acceptable for
> small-scale private use, but for large scale public repositories I'd
> say it's totally unacceptable.


That bug isn't about a collision, it's about files being replaced underneath
Packages feet.  Even with the tricks you've leveled the issue of things
changing under foot still is possible- you've just made the race less
likely.

What I was talking about was solving this issue once and for all via
restructuring, and specifically refering to the potential of an md5
collision in the URI space- specifically what I'm implementing for pkgcore
is the ability to do stupid stuff like this-

http://host/binpkg-store/$MD5.{txz,tbz2,tgz}

then have multiple views accessible just via pointing the binpkg repo remote
url at

http://host/views/license/oss-approved/
http://host/views/keywords/amd64/stable/
http://host/views/raw/ # no filtering on the view of the binpkg repo, see
everything.

Via restructuring where the binpkgs are stored and doing this approach,
multiple views can be had easily into the repo.  An additional benefit of
this approach is that via making URI able to point outside the host, you
could combine multiple seperate repositories into one just via a view.



> Eventually, I'd like to see gentoo
> officially distributing binary packages, so that we'll be able to
> get a slice of the binary distribution pie. When that happens, we're
> certainly not going to want to have race conditions like these in
> our public binhosts.
>

I'd suggest abandoning the current repository layout of Packages then, since
it's irrevocably flawed.  You can hack around it via jamming timestamp/md5
info into URI, but that's not a sane solution.

~harring