Re: [gentoo-user] Local portage git mirror

[Peter Humphrey]

On Tuesday, 17 January 2017 20:51:38 GMT Bill Kenworthy wrote:
> On 17/01/17 18:03, Peter Humphrey wrote:
> > Hello list,
> >
> > Does anyone have some pointers to setting up a local portage mirror for
> > git to serve? I'm looking at http://www.funtoo.org/Portage_Git_Mirror
> > but some details are obscure to me. I hope I'm not going to have to
> > roll my sleeves up and learn the whole of git; I just want to cut down
> > the time my boxes take to sync via git - some of them have their
> > portage trees nfs-mounted in chroots on my workstation and take
> > anything up to 20 minutes to sync.
> >
> > One question comes up straight away: should the git-served repository be
> > the host's own /usr/portage tree? And what cron task should I run to
> > refresh the mirror?
> >
> > Any help would be gratefully received.
> >
> > (I can still read e-mails via web-mail even when KMail is playing up.)
>
> I moved my systems to a local git repos some time ago - it works (fast -
> much faster than rsync!)but its not perfect:
>
> It would be nice if someone can share how it really should be done!

It seems to me to be ideal material for a wiki page. Any offers?

> Below is the contents of an email to the list in 2015 when I got it
> working:
>
> * problems:
> the files in /usr/share/portage sometimes get overridden and give an
> error message
> renaming "gentoo" to "olympus" (the server host) made sense at the time
> but creates odd errors at times requiring files to be restored.
> git doesnt use the portage checksums which besides odd errors at times
> required using "thin-manifests = true" in
> "/usr/portage/metadata/layout.conf"

Many thanks, Bill. That'll give me something to think about!

-- 
Regards
Peter

> ++++++++++++++++++
> Replace your rsync infrastructure with a git one.  After set up, there
> is faster syncing than rsync, both externally to the gentoo
> infrastructure (reasonable ADSL link), and really fast compared to rsync
> internally.  Not simple, but its a one off hit on just a few machines
> for me.  Works for me, no warranty etc.
>
> Thanks to Michael and Hasufell for the hints needed to get this working.
> Apologies for the line wrap.
>
>
>
> Replace with your own values!
> repo name: olympus
> repo master host: olympus.lan.localdomain
>
> On the master:
>
> install git !!!
>
> Download https://github.com/hasufell/portage-gentoo-git-config
>
> unzip
>
> cp -v portage-gentoo-git-config-master/etc/portage/repo.postsync.d/*
> /etc/portage/repo.postsync.d/
>
> cp portage-gentoo-git-config-master/etc/portage/repos.conf/gentoo.conf
> /etc/portage/repos.conf/
>
> if needed: chmod +x /etc/portage/repo.postsync.d/sync_overlay_cache
> cd /usr/
> mv portage portage.old
>
> emerge --sync
>
>
> To use as the master for your own mirror system:
> Uncomment the lines in /etc/rsyncd.conf to set up your own
> gentoo-portage rsync server - still needed for now
>
> rc-update add rsyncd
>
> /etc/init.d/rsyncd start
>
>
>
> On the internal clients wanting to use the mirror:
> reccomended: set up passwordless ssh between clients and the master
>
> emerge dev-vcs/git
>
> create /etc/portage/repos.conf/olympus.conf as follows
> --------------------
> [DEFAULT]
> main-repo = olympus
>
> [olympus]
> location = /usr/portage
> sync-type = git
> sync-uri = git://olympus.lan.localdomain:/usr/portage
> auto-sync = yes
> --------------------
>
> create /etc/portage/repo.postsync.d/sync_olympus
> --------------------
> #!/bin/bash
>
> repository_name="${1}"
> repository_path="${3}"
>
> [[ ${repository_name} == "olympus" ]] || exit 0
>
> source /lib/gentoo/functions.sh
>
> # Number of jobs for egencache, default is number or processors.
> parallel_jobs="$(nproc)"
>
> # caches
> ebegin "Fetching pre-generated metadata cache for ${repository_name}"
> 	rsync -a
> rsync://olympus.lan.localdomain/gentoo-portage/metadata/md5-cache/
> "${repository_path}"/metadata/md5-cache/
> eend $?
>
> ebegin "Updating metadata cache for ${repository_name}"
> 	egencache --jobs="${parallel_jobs}" --repo="${repository_name}"
> --update --update-use-local-desc
> eend $?
>
> #DTD's
> DTDDIR="${repository_path}"/metadata/dtd
> ebegin "Updating DTDs"
> if [[ -e ${DTDDIR} ]]; then
>         git -C "${DTDDIR}" pull -q --ff
> else
>         git clone olympus.lan.localdomain:/usr/portage/metadata/dtd
> "${DTDDIR}"
> fi
> eend "$?"
>
> #GLSA's
> GLSADIR="${repository_path}"/metadata/glsa
> ebegin "Updating GLSAs"
> if [[ -e ${GLSADIR} ]]; then
>         git -C "${GLSADIR}" pull -q --ff
> else
>         git clone olympus.lan.localdomain:/usr/portage/metadata/glsa
> "${GLSADIR}"
> fi
> eend "$?"
>
> # herds
> ebegin "Updating herds.xml"
>     rsync -a
> rsync://olympus.lan.localdomain/gentoo-portage/metadata/herds.xml
> "${repository_path}"/metadata/md5-cache/
> eend $?
>
> #news
> NEWSDIR="${repository_path}"/metadata/news
> ebegin "Updating news items"
> if [[ -e ${NEWSDIR} ]]; then
>         git -C "${NEWSDIR}" pull -q --ff
> else
>         git clone olympus.lan.localdomain:/usr/portage/metadata/news
> "${NEWSDIR}"
> fi
> eend $? "Try to remove ${NEWSDIR}"
> --------------------
>
>
> mv /usr/portage /usr/portage.old
>
> git clone olympus.lan.localdomain:/usr/portage /usr/portage
>
> echo 'olympus' > /usr/portage/profiles/repo_name
>
> echo 'masters = olympus' > /usr/local/portage/metadata/layout.conf
>
> echo 'thin-manifests = true' >> /usr/local/portage/metadata/layout.conf
>
> cp  /usr/portage/metadata/layout.conf
> /usr/local/portage/metadata/layout.conf
>
> cp /etc/portage/repos.conf/olympus.conf
> /usr/share/portage/config/repos.conf
>
> emerge --sync
> emerge --sync
>
> * the first sync will complete the process, the second is to check there
> are no errors.