Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Michael Mol]

On Fri, Nov 30, 2012 at 10:57 AM, Richard Yao <ryao@g.o> wrote:
> On 11/28/2012 11:08 AM, Matthew Thode wrote:
>> On 11/28/2012 09:05 AM, Richard Yao wrote:
>>> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
>>>> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ryao@g.o> wrote:
>>>>> We could slightly simplify the handbook installation procedure if we
>>>>> told people to use emerge-webrsync to fetch the initial snapshot.
>>>>
>>>> Using emerge-webrsync also makes the installation process more robust,
>>>> since it only requires HTTP access (whereas many firewalls restrict
>>>> RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
>>>> that it should be the primary recommended portage tree synchronization
>>>> method.
>>>>
>>>
>>> The only downside of which I am aware is increased network traffic.
>>> However, we could redesign emerge-webrsync to take advantage of GNU
>>> Tar's incremental archive functionality.
>>>
>>> That would permit us to mirror compressed diffs in addition to regular
>>> portage snapshots. Doing that well could reduce bandwidth requirements.
>>>
>> weekly fulls and daily diffs?
>>
>
> Determining what is right here probably requires calculus, but this
> scheme does not seem like a bad choice to me. My main concern is that
> maintaining weekly full snapshots would require too much space for the
> mirrors. It might be better go monthly, with diffs on the following
> intervals:
>
> 1 week
> 1 day
> 30 minutes
>
> Doing that would eliminate the benefit of rsync entirely, with the
> caveat that we now need to mirror a ton of diffs. This would make it
> easy for us to provide the ability to obtain historical snapshots, which
> would be nice.

Worth noting that all this moves us nicely in the direction of
allowing HTTP proxies to cache data, reducing load on mirrors. And
moves us in the direction of implementing mirrors themselves as a
network of caching proxies.

--
:wq