Re: [gentoo-user] speedup remote portage

[Rich Freeman]

On Thu, Feb 27, 2020 at 2:27 PM james <garftd@×××××××.net> wrote:
>
> On 2/26/20 7:08 PM, William Kenworthy wrote:
> > Hi,
> >
> >  ��� due to space considerations on my laptop I have moved portage onto a
> > network share (moosfs, mfsmounted) - slower but works fine.� However,
> > being a laptop trying to update/install when out and about is both very
> > slow and network intensive through a vpn - again, it works but is even
> > slower to the point of not always being practical
> >
> > Is there a way to localise/speedup portage scanning parts of the
> > update/install process?
>
>
> A simpler, much less sophisticated update, what I do is
> use emerge -f option to 'fetch only' option first. The selectively
> update; or you can use a usb-3.1+ device for fast easy upgrades, due to
> laptop limitations, but the communications data channel limitations
> leave you at the mercy of the available wireless bandwidth characteristics.

So, a few thoughts here:

1.  Definitely make sure you're splitting out distfiles.  Any new
install does this already, but mixing up the tree and distfiles is
mixing very different types of data with different requirements.
2.  Consider whether you REALLY need to share all this stuff.  The
whole tree isn't THAT big, so it might be easier to just locally sync.
If network bandwidth is an issue, consider setting up a local mirror
so that you sync (and fetch distfiles) within your network when
available, and then just sync directly from the internet when you're
remote.
3.  If you REALLY need that portage tree to be on the network,
consider keeping just the metadata directory local.  That can be
generated on-demand or synced, but either way it needs to be updated
anytime the repo itself is updated.  Generally with rsync we
distribute both together.  I suspect that having local metadata will
cut down on your network IO when you're doing dep resolution/etc.
4.  Squashfs was mentioned.  In theory that is a nice compact
solution.  In practice you might find it to be a pain to update since
it is read-only.  But you could update your network tree, then create
a squashfs and stick that on your network, and then sync the squashfs
file for local mounting.

I'm not sure how well moosefs performs for this sort of thing in
general.  I personally use lizardfs to store large multimedia files
and it works fine, but streaming multi-GB multimedia that ends up
getting buffered is a very different access pattern than sourcing a
bazillion ebuild scripts.  I'd be interested in how you're liking
moosefs for this sort of thing in general.  For accessing tons of
little files I think all your latencies matter, but in particular you
want low latency to your master server as that is going to end up
limiting every read/open you perform.  You might be able to play with
the cache settings on the client side, though at least in lizardfs
I've found some of those to be buggy - they're generally passed as
mount options.

In the past I've had one main Gentoo box sync its repo from the web,
and had other hosts in the LAN either bind-mount this (containers), or
sync this over the network.  Likewise I generally share my main host
distfiles as a mirror which is preferentially used by all my other
hosts - you don't get a 100% hit rate and the main host won't get a
copy of missed files, but it probably cuts down on 80+% of the network
fetching as long as the main host does its fetches before the other
hosts do.  If the main host lacks a distfile then the MIRRORS setting
has a regular Gentoo mirror listed next.

Honestly, unless you're running a completely private repo for whatever
reason I would not try to go remotely mounting your repo over the
internet from your moosefs server.  I bet you could fetch anything
needed faster from a regular internet mirror in that case, and if
portage could use mirrors seamlessly over internet latencies chances
are we'd all be doing that already.  Pretty much every distro
maintains a local repo on every host for this reason.

-- 
Rich