Re: [gentoo-soc] Improved binary package support

[Ethan Hall]

Would getting hashes to agree be a hard thing to accomplish?  If you use the
binary build feature in portage you could save the build in a compressed
format.  Then have a hash generate set up where it would take the name,
version, and use flags, cflags and hash just that information. Have a
program that would run as a deamond on compter it could update themselves to
something like a torrent tracker.  The torrent name could the the hashed
values.  Then we could have the data that would need to be installed passed
via torrent to all other users. It would be an interesting implementation.
Also you could use a DC++ idea and have a hub set up where you could only
share compiled programs?

This would make the idea distributed so it would not have to have ~10451G of
space used on any single server.  People would host 'packages' as they
compiled them.  The more comon the program was, the faster the downloads
would be.

Another use would be releasing common programs/sets as packages? Example,
Xorg. There are tons of programs that go along with it.  If you host
commonly used programs on a service then people could download the program
and use it while they compiled it with their own use flags.  We could offer
programs with all flags enabled, the package manager would go get the
prebuild program, keep track of all the files and then start the custom
build according to the users own settings.  After the build is done, before
it is moved from its sandbox, purdge the old files from the downloaded,
pre-compiled code, and then move the files from the sand box.

On Tue, Mar 24, 2009 at 2:22 AM, <mmacleod@××××××××××.za> wrote:

> On Tuesday 24 March 2009 06:41:14 Alec Warner wrote:
> > On Mon, Mar 23, 2009 at 4:24 PM, Philipp Riegger <lists@××××××××××××.de>
> wrote:
> > > On Mon, 23 Mar 2009 08:52:38 +0200
> > >
> > > mmacleod@××××××××××.za wrote:
> > >><snip>
> > > I'm not sure, if this is the right way to go. I'd prefer some kind of
> > > build server with better configuration and management than exists at
> > > the moment.
> At a very rough calculation for just one build server to hold all packages
> for
> one architecture with only one use flag combination you are looking at
> ~0451
> gigabytes of space. This can be reduced somewhat with difference patches
> but
> then you are stuck spending vast amount of compute times on the differences
> and still need a fairly vast amount of hard drive space.
> I don't even know how to venture a guess at the compile time for this at
> the
> moment but I am fairly certain it is completely impractical for now.
> The best that could be achieved with a nice build server would be binaries
> for
> a small subset of what is in portage. This assumes a good dedicated build
> server is available for such a thing.
>
> My idea can function with no build server, and then incorporate build
> servers
> as they come along so it can take advantage of both.
>
> In an ideal world Gentoo will have enough build servers to generate them
> all
> internally and maybe we will get there one day, this idea is scalable so it
> could continue to function with just build servers eventually, however I
> suspect that we are a long way away from having build server capable of
> this.
> Feel free to correct me if I am wrong on this though.
> > > I'm not sure, if the p2p network is trustworthy enough,
> > > even with your hash stuff. What if a user creates a package with some
> > > backdoor in it? How would you notice that?
> This is just a simple example for now, actual security levels could be more
> complex to be even more effective.
> Imagine however the following three security levels.
> 1) Hash contributed by 100 users
> 2) Hash contributed by 1000 users
> 3) Hash verified by build server(build server verifies packages from level
> 2
> based on popularity and/or a list of "important" packages to prefer)
>
> Now if a user creates a package with a back door he will have to trick or
> collaborate with 100 other users in order to get the people on level 1
> security setting to download his modified sources and compile them all
> without
> 1 other legit user having a clashing hash, the instant there is a clash his
> package will be flagged and action taken.
> This is still possible I admit but highly unlikely, especially if users
> acting
> as "compile hosts" are randomly selected to verify such packages.
>
> At level 2 security the same thing except now 1000 users all have to agree
> on
> the package hash.
> Even more unlikely.
>
> At level 3 the package has been compiled by build server itself so the only
> way the package can contain a back door is if the server has been
> compromised
> or the original sources have been compromised. In this case it is no less
> secure then our current set up where everyone compiles as it would be
> equally
> hard to compromise the build server as to get a tainted tarball into the
> official portage tree.
>
> The levels above are just examples, the proper scheme would likely contain
> a
> few more levels as well as a bit more complexity, however it needs more
> thought first and I don't want to get too complex yet, just trying to show
> the
> basic idea.
>
> I am sure many users would be happy enough with even level 1 security
> packages, I would probably not hesitate to have such packages on my
> machine.
> For the more paranoid they can wait a bit longer for packages to reach a
> higher level or compile it themselves in the meantime.
> > >> <snip>
> > >
> > > Not sure, if this was already discussed: Usually, software has
> > > versions. Gentoo adds revisions (these -rX endings) to have different
> > > ebuild versions for the same software version. Now, for binary packages
> > > we would need another revision-like addition (because there can be
> > > pkg-foo-1.2.3 linked against lib-bar:1 and linked against lib-bar:2).
> The "Improved binary package support" idea on the wiki as well as the
> related
> bugzilla thread already cover this as well as possible ways to deal with
> it.
> > > If I were the buildserver, I would build the new libbar once it is in
> > > the tree and then use revdep-rebuild and emerge @preserved-rebuild to
> > > figure out, what needs to be rebuilt. Then I would rebuild those
> > > packages and increase the, let's call it binary-revision. I'd repeat
> > > that until all was fine (which should be the case after 1 iteration,
> > > most of the time).
>
> > > Another thing: Smaller installations. Maybe you could work on moving
> > > more software out of @system, that for example gcc does not need to be
> > > installed on binary-only hosts. Maybe you could create some
> > > INSTALL_MASK files that would make the system smaller leaving out
> > > headers, maybe static libraries and stuff like that. What would you
> > > think about that one?
> > I think "binary only profile" ?
> A binary profile would definitely be nice, I don't think it is worth
> considering until binaries for most things are largely available first
> though.
>
>
> Another nice spin off of having these binaries would be improved multilib
> support, portage would need some improvement to handle this, but a 64 bit
> user(for example) could have portage get a 32 bit binary for any package
> and
> run it instead of needing to rely on the "*-bin" packages and/or chroot
> when
> no -bin is available.
>
>


-- 
Ethan Hall
ekhall@××××××.edu
ethankhall@×××××.com