Re: [gentoo-user] speedup remote portage

[William Kenworthy]

Hi Rich, I will experiment with a local metadata - that seems like it
might give a worthwhile speedup (its currently about 910Mb so not too big)

Bill K.


On 28/2/20 3:44 am, Rich Freeman wrote:
> On Thu, Feb 27, 2020 at 2:27 PM james <garftd@×××××××.net> wrote:
>> On 2/26/20 7:08 PM, William Kenworthy wrote:
>>> Hi,
>>>
>>>  ��� due to space considerations on my laptop I have moved portage onto a
>>> network share (moosfs, mfsmounted) - slower but works fine.� However,
>>> being a laptop trying to update/install when out and about is both very
>>> slow and network intensive through a vpn - again, it works but is even
>>> slower to the point of not always being practical
>>>
>>> Is there a way to localise/speedup portage scanning parts of the
>>> update/install process?
>>
>> A simpler, much less sophisticated update, what I do is
>> use emerge -f option to 'fetch only' option first. The selectively
>> update; or you can use a usb-3.1+ device for fast easy upgrades, due to
>> laptop limitations, but the communications data channel limitations
>> leave you at the mercy of the available wireless bandwidth characteristics.
> So, a few thoughts here:
>
> 1.  Definitely make sure you're splitting out distfiles.  Any new
> install does this already, but mixing up the tree and distfiles is
> mixing very different types of data with different requirements.
> 2.  Consider whether you REALLY need to share all this stuff.  The
> whole tree isn't THAT big, so it might be easier to just locally sync.
> If network bandwidth is an issue, consider setting up a local mirror
> so that you sync (and fetch distfiles) within your network when
> available, and then just sync directly from the internet when you're
> remote.
> 3.  If you REALLY need that portage tree to be on the network,
> consider keeping just the metadata directory local.  That can be
> generated on-demand or synced, but either way it needs to be updated
> anytime the repo itself is updated.  Generally with rsync we
> distribute both together.  I suspect that having local metadata will
> cut down on your network IO when you're doing dep resolution/etc.
> 4.  Squashfs was mentioned.  In theory that is a nice compact
> solution.  In practice you might find it to be a pain to update since
> it is read-only.  But you could update your network tree, then create
> a squashfs and stick that on your network, and then sync the squashfs
> file for local mounting.
>
> I'm not sure how well moosefs performs for this sort of thing in
> general.  I personally use lizardfs to store large multimedia files
> and it works fine, but streaming multi-GB multimedia that ends up
> getting buffered is a very different access pattern than sourcing a
> bazillion ebuild scripts.  I'd be interested in how you're liking
> moosefs for this sort of thing in general.  For accessing tons of
> little files I think all your latencies matter, but in particular you
> want low latency to your master server as that is going to end up
> limiting every read/open you perform.  You might be able to play with
> the cache settings on the client side, though at least in lizardfs
> I've found some of those to be buggy - they're generally passed as
> mount options.
>
> In the past I've had one main Gentoo box sync its repo from the web,
> and had other hosts in the LAN either bind-mount this (containers), or
> sync this over the network.  Likewise I generally share my main host
> distfiles as a mirror which is preferentially used by all my other
> hosts - you don't get a 100% hit rate and the main host won't get a
> copy of missed files, but it probably cuts down on 80+% of the network
> fetching as long as the main host does its fetches before the other
> hosts do.  If the main host lacks a distfile then the MIRRORS setting
> has a regular Gentoo mirror listed next.
>
> Honestly, unless you're running a completely private repo for whatever
> reason I would not try to go remotely mounting your repo over the
> internet from your moosefs server.  I bet you could fetch anything
> needed faster from a regular internet mirror in that case, and if
> portage could use mirrors seamlessly over internet latencies chances
> are we'd all be doing that already.  Pretty much every distro
> maintains a local repo on every host for this reason.
>