Re: [gentoo-portage-dev] Package compression header for binhosts

[Zac Medico]

On 06/01/2010 02:52 PM, Brian Harring wrote:
> That bug isn't about a collision, it's about files being replaced underneath
> Packages feet.  Even with the tricks you've leveled the issue of things
> changing under foot still is possible- you've just made the race less
> likely.

AFAIK the race is completely eliminated by the RCU-like snapshot
mechanism. I think I like your hash-in-the-filename idea better
though, since it seems simpler to implement and maintain. It can
even be done with the existing version 0 format by abusing the
per-package PATH attribute to refer to a filename that contains a
hash (maybe something like $CATEGORY/$PF.$MD5.tbz2). I wouldn't want
to abuse PATH for compression in version 0 though, since clients are
only required to support tbz2.

> What I was talking about was solving this issue once and for all via
> restructuring, and specifically refering to the potential of an md5
> collision in the URI space- specifically what I'm implementing for pkgcore
> is the ability to do stupid stuff like this-
>
> http://host/binpkg-store/$MD5.{txz,tbz2,tgz}

That would be the MD5 of the entire file, after compression and
having the xpak segment appended, right?

> then have multiple views accessible just via pointing the binpkg repo remote
> url at
>
> http://host/views/license/oss-approved/
> http://host/views/keywords/amd64/stable/
> http://host/views/raw/ # no filtering on the view of the binpkg repo, see
> everything.

So, the default path of a package would come from looking at the MD5
in the Packages file and then mapping that to a path?

> Via restructuring where the binpkgs are stored and doing this approach,
> multiple views can be had easily into the repo.  An additional benefit of
> this approach is that via making URI able to point outside the host, you
> could combine multiple seperate repositories into one just via a view.

This might also be useful for creating per-profile views while
allowing packages to be shared between profiles in cases when
hosting a separate build would be redundant. It might be possible to
save lots of build time, disk space, and testing that way.

Being able to have multiple builds of the same package with
different USE settings is also solves bug 150031 [1].

>> Eventually, I'd like to see gentoo
>> officially distributing binary packages, so that we'll be able to
>> get a slice of the binary distribution pie. When that happens, we're
>> certainly not going to want to have race conditions like these in
>> our public binhosts.
>>
>
> I'd suggest abandoning the current repository layout of Packages then, since
> it's irrevocably flawed.  You can hack around it via jamming timestamp/md5
> info into URI, but that's not a sane solution.

Shrug, it's a handy way to solve race conditions given the existing
version 0 format. It's not optimal, so we'll surely want something
better in version 1.

[1] http://bugs.gentoo.org/show_bug.cgi?id=150031
-- 
Thanks,
Zac