Re: [gentoo-user] Open Question: The feasibility of a complete portage binhost

[Alec Ten Harmsel]

On 01/21/2015 07:47 AM, Sam Bishop wrote:
> So I've been thinking crazy thoughts.
>
> Theoretically it can't be that hard to do a complete package binhost for gentoo.

I love that you qualify this with "theoretically."

>
> To be clear, when i say complete, Im referring to building, all
> versions of all ebuilds marked stable or unstable on amd64, with every
> combination of use flags.

Every ebuild with every combination of USE flags? This is likely
impossible, and definitely not feasible. With 17000ish ebuilds in the
portage tree and assuming each only has 2 USE flags, this would be
building 17000*2^2 = 68,000 packages. If average build time is 20
seconds (nice server w/ SSD and enough RAM to build in /tmp), it'd take
377ish hours to do an initial build of the tree. I guess this isn't so
bad. Of course, there are outliers like www-client/firefox: 19
non-language USE flags, so 2^19 different firefox permutations at a fast
5 minutes apiece would take 43000 hours. I haven't looked at
REQUIRED_USE, so there could be less than 2^19 different combinations of
flags; taking it down to 2^10 combinations is only 85 hours or so.

>
> This pretty much boils down to bytes and bytes of storage + compute
> resources. Both of which are easily available to me. So I began
> pondering and here I am, thinking to myself "is this really all there
> is too it"?

A full CentOS mirror is ~600GB iirc, so you're gonna need a ton of storage.

> Does it really come down to CPU cycles and repeatedly running through
> the following commands for each combination of ebuild, version and use
> flags
>   emerge --emptytree --onlydeps ${name}
>   emerge --emptytree --buildpkgonly --buildpkg ${name}
>
> Obviously running them in a clean environment each time, either by
> chroot or other means.
> Then just storing the giant binhost somewhere suitable such as an AWS
> s3 bucket setup to work via HTTP so the normal tools work fine with
> it.
>

I haven't used binpkgs in a long time, but I think USE on the client
machine has to match the USE of the package being installed. Managing
all of this would be a nightmare unless you wrote your own special
portage server that served up binpkgs in a USE-aware way and a portage
host could request a binpkg with a certain USE.

Theoretically, great idea. I think this would be possible if you had
maybe 3 or 4 different USE combos (i.e. one for servers, one for KDE
client machines, one for gnome clients, etc.).

Alec

P.S. I'm reasonably sure my math is correct, but I would appreciate
corrections.