Re: [gentoo-soc] Improved binary package support

On Tuesday 24 March 2009 06:41:14 Alec Warner wrote:
> On Mon, Mar 23, 2009 at 4:24 PM, Philipp Riegger <lists@××××××××××××.de>
wrote:
> > On Mon, 23 Mar 2009 08:52:38 +0200
> >
> > mmacleod@××××××××××.za wrote:
> >><snip>
> > I'm not sure, if this is the right way to go. I'd prefer some kind of
> > build server with better configuration and management than exists at
> > the moment.
At a very rough calculation for just one build server to hold all packages for 
one architecture with only one use flag combination you are looking at ~0451 
gigabytes of space. This can be reduced somewhat with difference patches but 
then you are stuck spending vast amount of compute times on the differences 
and still need a fairly vast amount of hard drive space.
I don't even know how to venture a guess at the compile time for this at the 
moment but I am fairly certain it is completely impractical for now.
The best that could be achieved with a nice build server would be binaries for 
a small subset of what is in portage. This assumes a good dedicated build 
server is available for such a thing.

My idea can function with no build server, and then incorporate build servers 
as they come along so it can take advantage of both.

In an ideal world Gentoo will have enough build servers to generate them all 
internally and maybe we will get there one day, this idea is scalable so it 
could continue to function with just build servers eventually, however I 
suspect that we are a long way away from having build server capable of this.
Feel free to correct me if I am wrong on this though.
> > I'm not sure, if the p2p network is trustworthy enough,
> > even with your hash stuff. What if a user creates a package with some
> > backdoor in it? How would you notice that?
This is just a simple example for now, actual security levels could be more 
complex to be even more effective.
Imagine however the following three security levels.
1) Hash contributed by 100 users
2) Hash contributed by 1000 users
3) Hash verified by build server(build server verifies packages from level 2 
based on popularity and/or a list of "important" packages to prefer)

Now if a user creates a package with a back door he will have to trick or 
collaborate with 100 other users in order to get the people on level 1 
security setting to download his modified sources and compile them all without 
1 other legit user having a clashing hash, the instant there is a clash his 
package will be flagged and action taken.
This is still possible I admit but highly unlikely, especially if users acting 
as "compile hosts" are randomly selected to verify such packages.

At level 2 security the same thing except now 1000 users all have to agree on 
the package hash. 
Even more unlikely.

At level 3 the package has been compiled by build server itself so the only 
way the package can contain a back door is if the server has been compromised 
or the original sources have been compromised. In this case it is no less 
secure then our current set up where everyone compiles as it would be equally 
hard to compromise the build server as to get a tainted tarball into the 
official portage tree.

The levels above are just examples, the proper scheme would likely contain a 
few more levels as well as a bit more complexity, however it needs more 
thought first and I don't want to get too complex yet, just trying to show the 
basic idea.

I am sure many users would be happy enough with even level 1 security 
packages, I would probably not hesitate to have such packages on my machine. 
For the more paranoid they can wait a bit longer for packages to reach a 
higher level or compile it themselves in the meantime.
> >> <snip>
> >
> > Not sure, if this was already discussed: Usually, software has
> > versions. Gentoo adds revisions (these -rX endings) to have different
> > ebuild versions for the same software version. Now, for binary packages
> > we would need another revision-like addition (because there can be
> > pkg-foo-1.2.3 linked against lib-bar:1 and linked against lib-bar:2).
The "Improved binary package support" idea on the wiki as well as the related 
bugzilla thread already cover this as well as possible ways to deal with it.
> > If I were the buildserver, I would build the new libbar once it is in
> > the tree and then use revdep-rebuild and emerge @preserved-rebuild to
> > figure out, what needs to be rebuilt. Then I would rebuild those
> > packages and increase the, let's call it binary-revision. I'd repeat
> > that until all was fine (which should be the case after 1 iteration,
> > most of the time).

> > Another thing: Smaller installations. Maybe you could work on moving
> > more software out of @system, that for example gcc does not need to be
> > installed on binary-only hosts. Maybe you could create some
> > INSTALL_MASK files that would make the system smaller leaving out
> > headers, maybe static libraries and stuff like that. What would you
> > think about that one?
> I think "binary only profile" ?
A binary profile would definitely be nice, I don't think it is worth 
considering until binaries for most things are largely available first though.


Another nice spin off of having these binaries would be improved multilib 
support, portage would need some improvement to handle this, but a 64 bit 
user(for example) could have portage get a 32 bit binary for any package and 
run it instead of needing to rely on the "*-bin" packages and/or chroot when 
no -bin is available.