Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Alec Warner]

On Fri, Nov 30, 2012 at 8:59 AM, Ian Stakenvicius <axs@g.o> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 30/11/12 11:15 AM, Michael Mol wrote:
>> On Fri, Nov 30, 2012 at 10:57 AM, Richard Yao <ryao@g.o>
>> wrote:
>>> On 11/28/2012 11:08 AM, Matthew Thode wrote:
>>>> On 11/28/2012 09:05 AM, Richard Yao wrote:
>>>>> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
>>>>>> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao
>>>>>> <ryao@g.o> wrote:
>>>>>>> We could slightly simplify the handbook installation
>>>>>>> procedure if we told people to use emerge-webrsync to
>>>>>>> fetch the initial snapshot.
>>>>>>
>>>>>> Using emerge-webrsync also makes the installation process
>>>>>> more robust, since it only requires HTTP access (whereas
>>>>>> many firewalls restrict RSYNC). Besides, emerge-webrsync
>>>>>> can check PGP signatures, so I think that it should be the
>>>>>> primary recommended portage tree synchronization method.
>>>>>>
>>>>>
>>>>> The only downside of which I am aware is increased network
>>>>> traffic. However, we could redesign emerge-webrsync to take
>>>>> advantage of GNU Tar's incremental archive functionality.
>>>>>
>>>>> That would permit us to mirror compressed diffs in addition
>>>>> to regular portage snapshots. Doing that well could reduce
>>>>> bandwidth requirements.
>>>>>
>>>> weekly fulls and daily diffs?
>>>>
>>>
>>> Determining what is right here probably requires calculus, but
>>> this scheme does not seem like a bad choice to me. My main
>>> concern is that maintaining weekly full snapshots would require
>>> too much space for the mirrors. It might be better go monthly,
>>> with diffs on the following intervals:
>>>
>>> 1 week 1 day 30 minutes
>>>
>>> Doing that would eliminate the benefit of rsync entirely, with
>>> the caveat that we now need to mirror a ton of diffs. This would
>>> make it easy for us to provide the ability to obtain historical
>>> snapshots, which would be nice.
>>
>> Worth noting that all this moves us nicely in the direction of
>> allowing HTTP proxies to cache data, reducing load on mirrors. And
>> moves us in the direction of implementing mirrors themselves as a
>> network of caching proxies.
>>
>
> Idea makes sense, I wonder if implementation would be better served by
> leveraging the fact that we already produce daily full snapshots:
>
> 1 - continue to provide the daily snapshots we do now
> 2 - provide two weeks (more?) of daily diffs, such that a daily
> snapshot from up to two weeks ago can be updated to present day
> 3 - provide hourly or 30-minute update diffs to get latest changes.
>
> If the tree is more than two weeks old, emerge-webrsync would just
> grab the latest daily plus the hourly diffs.
>
> If the tree is less than two weeks old, grab the daily diffs and
> hourly diffs.  The local copy of the tree itself would need to be
> rolled back to the best-available daily diff before these diff updates
> could be applied; this may mean that a local cache of the latest
> full-day snapshot needs to be kept and/or generated.  Also if said
> cache doesn't exist, then the whole full-day snapshot would be grabbed.
>
> The advantage to this would be significantly fewer distfiles, although
> the logic in emerge-webrsync would possibly be more complex.
>
> Regarding rolling back the local tree to a known-good state, I think
> that would be required regardless of the method as any local changes
> made to the tree by users would need to be discarded, right?

How about we not change the docs until someone eagerly implements all
the stuff you just said?

Note that from an infra POV our existing system works fairly well and
requires no day-to-day tinkering.
I'm always happy to consider new options, but work needs to put in to
make it feasible. I'm sure if we switched to http with zsync or
something, we could make it feasible. I want to see a prototype, etc.

-A

>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
>
> iF4EAREIAAYFAlC45f0ACgkQ2ugaI38ACPChKgD9GOBptQ9jJ1/eYyq1NEl5Oq1E
> dVy9UOab80bG5FZB9LwBAKwsifnT+iE3n/4d/ljnuT2qCnbtXNYr7yBjF/VcEpkq
> =y9eB
> -----END PGP SIGNATURE-----
>