Re: [gentoo-server] Pre-GLEP: Speed up Portage updates with static baseline tree and updates overlays

[Vladimir G. Ivanovic]

On Thu, 2006-09-07 at 15:30 -0400, Andy Dustman wrote:
> gentoo-server is probably not the best list for this discussion, but I
> suspect many or most of the people on it will have an interest in
> this, so it is somewhat-on-topic. If there's any interest/traction,
> then I'll redo this as a GLEP. If you like the idea but think it
> should be discussed off this list, contact me directly.
>
> There are two main issues with the Portage tree as I see it:
>
> 1) There's no way to get just security updates.

I run this script every hour:

        #!/bin/sh
        glsa-check -f new 2>/dev/null
        [[ $? -eq 0 ]] || echo "glsa-check: error"

Does this not do what you're looking for?

>
> 2) Even with recent metadata cache update improvements, it still takes
> a long time, and a lot of resources, to sync the tree.

I do my updates early in the morning when I'm asleep, so I don't really
care how long they take (as long as it's less than 4 hours).

Why does update speed matter to you?

--- Vladimir

>
> Now while I firmly believe Gentoo has the best build system and
> packagement management bar none, I think there are still a few good
> ideas out there in other systems to be borrowed/stolen.
>
> Currently, emerge --sync (typically) does an rsync off one of the
> Gentoo rsync mirrors and then updates the metadata cache. This process
> involves examining every file in the Portage tree on both sides of the
> connection (stat). This is fixed overhead which is linearly
> proportional to the number of files in the tree, regardless of how
> many actual updates there may be. Fortunatelly, there is an
> optimization that checks a specific timestamp file on the mirror and
> skips the real sync if it matches the local timestamp.
>
> Compare this with the Debian/Ubuntu method (I'll follow Ubuntu's
> conventions, since I'm more familiar with it): Each time there is is a
> new release, you get a new dist, plus dist-updates, and dist-security,
> and a couple others I'll ignore for now. For example, Ubuntu's current
> release is dapper, so there is also dapper-updates and
> dapper-security.
>
> Now the beauty of this is that once a release is actually final, the
> main release files (i.e. dapper) *do not change*; any updated package
> specs go into the -updates or -security directories. If most of the
> packages do not change during the update life of a distribution, i.e.
> updates are a relatively small percentage, then this is a big win.
> (AFAIK, apt-get and other methods check remote timestamps before
> downloading.)
>
> How can this idea be applied to Portage? In this sort of scheme, you'd
> have something analogous to an apt.sources line in Debian in profile's
> make.defaults:
>
> SYNC="rsync://rsync.gentoo.org/gentoo-releases"
> RELEASE="2006.1"
> RELEASE_OVERLAYS="updates security"
>
> Doing a emerge --sync would then perform the following:
>
> * rsync://rsync.gentoo.org/gentoo-releases/2006.1 to $PORTDIR
> * rsync://rsync.gentoo.org/gentoo-releases/2006.1-updates to $PORTDIR-updates
> * rsync://rsync.gentoo.org/gentoo-releases/2006.1-security to $PORTDIR-security
>
> $PORTDIR-updates and $PORTDIR-security could then be treated as
> implicit PORTDIR_OVERLAYs. SYNC could be overridden in /etc/make.conf
> as it is now.  If you wanted only security updates, then you could set
> RELEASE_OVERLAYS="security" in make.conf.
>
> This is now three trees to sync against instead of one, but the
> important feature is that the primary tree is now static data, once it
> is done as a final release, so there is only the timestamp check; and
> the other two trees should be relatively small. Obsolete ebuilds need
> only be removed when there is a new release, and this happens in a
> different tree. Potentially, only packages with at least one stable
> arch flag could go in updates; anything that is all ~arch or masked
> could go into a separate testing overlay.
>
> Known Issues:
>
> * Gentoo infrastructure changes are going to be required, particularly
> in the mirror system and repoman and the Portage CVS tree. However, it
> should not require major surgery on Portage/emerge.
>
> * End-users will have to change their configuration a bit, or at least
> SYNC. Note that I changed the SYNC location from gentoo-portage to
> gentoo-releases, since releases would need to be rooted outside the
> existing tree (resulting disaster left as an exercise for the reader).
> The existing tree may still be useful to keep for developers, testers,
> and good old backwards-compatibility (set RELEASE="").
>
> --
> This message has been scanned for memes and
> dangerous content by MindScanner, and is
> believed to be unclean.
> --
> gentoo-server@g.o mailing list
>
-- 
Vladimir G. Ivanovic <vgivanovic@×××××××.net>

-- 
gentoo-server@g.o mailing list