1 |
On 17/01/17 18:03, Peter Humphrey wrote: |
2 |
> Hello list, |
3 |
> |
4 |
> Does anyone have some pointers to setting up a local portage mirror for git |
5 |
> to serve? I'm looking at http://www.funtoo.org/Portage_Git_Mirror but some |
6 |
> details are obscure to me. I hope I'm not going to have to roll my sleeves |
7 |
> up and learn the whole of git; I just want to cut down the time my boxes |
8 |
> take to sync via git - some of them have their portage trees nfs-mounted in |
9 |
> chroots on my workstation and take anything up to 20 minutes to sync. |
10 |
> |
11 |
> One question comes up straight away: should the git-served repository be the |
12 |
> host's own /usr/portage tree? And what cron task should I run to refresh the |
13 |
> mirror? |
14 |
> |
15 |
> Any help would be gratefully received. |
16 |
> |
17 |
> (I can still read e-mails via web-mail even when KMail is playing up.) |
18 |
> |
19 |
|
20 |
I moved my systems to a local git repos some time ago - it works (fast - |
21 |
much faster than rsync!)but its not perfect: |
22 |
|
23 |
It would be nice if someone can share how it really should be done! |
24 |
Below is the contents of an email to the list in 2015 when I got it working: |
25 |
|
26 |
* problems: |
27 |
the files in /usr/share/portage sometimes get overridden and give an |
28 |
error message |
29 |
renaming "gentoo" to "olympus" (the server host) made sense at the time |
30 |
but creates odd errors at times requiring files to be restored. |
31 |
git doesnt use the portage checksums which besides odd errors at times |
32 |
required using "thin-manifests = true" in |
33 |
"/usr/portage/metadata/layout.conf" |
34 |
|
35 |
|
36 |
|
37 |
|
38 |
++++++++++++++++++ |
39 |
Replace your rsync infrastructure with a git one. After set up, there |
40 |
is faster syncing than rsync, both externally to the gentoo |
41 |
infrastructure (reasonable ADSL link), and really fast compared to rsync |
42 |
internally. Not simple, but its a one off hit on just a few machines |
43 |
for me. Works for me, no warranty etc. |
44 |
|
45 |
Thanks to Michael and Hasufell for the hints needed to get this working. |
46 |
Apologies for the line wrap. |
47 |
|
48 |
|
49 |
|
50 |
Replace with your own values! |
51 |
repo name: olympus |
52 |
repo master host: olympus.lan.localdomain |
53 |
|
54 |
On the master: |
55 |
|
56 |
install git !!! |
57 |
|
58 |
Download https://github.com/hasufell/portage-gentoo-git-config |
59 |
|
60 |
unzip |
61 |
|
62 |
cp -v portage-gentoo-git-config-master/etc/portage/repo.postsync.d/* |
63 |
/etc/portage/repo.postsync.d/ |
64 |
|
65 |
cp portage-gentoo-git-config-master/etc/portage/repos.conf/gentoo.conf |
66 |
/etc/portage/repos.conf/ |
67 |
|
68 |
if needed: chmod +x /etc/portage/repo.postsync.d/sync_overlay_cache |
69 |
cd /usr/ |
70 |
mv portage portage.old |
71 |
|
72 |
emerge --sync |
73 |
|
74 |
|
75 |
To use as the master for your own mirror system: |
76 |
Uncomment the lines in /etc/rsyncd.conf to set up your own |
77 |
gentoo-portage rsync server - still needed for now |
78 |
|
79 |
rc-update add rsyncd |
80 |
|
81 |
/etc/init.d/rsyncd start |
82 |
|
83 |
|
84 |
|
85 |
On the internal clients wanting to use the mirror: |
86 |
reccomended: set up passwordless ssh between clients and the master |
87 |
|
88 |
emerge dev-vcs/git |
89 |
|
90 |
create /etc/portage/repos.conf/olympus.conf as follows |
91 |
-------------------- |
92 |
[DEFAULT] |
93 |
main-repo = olympus |
94 |
|
95 |
[olympus] |
96 |
location = /usr/portage |
97 |
sync-type = git |
98 |
sync-uri = git://olympus.lan.localdomain:/usr/portage |
99 |
auto-sync = yes |
100 |
-------------------- |
101 |
|
102 |
create /etc/portage/repo.postsync.d/sync_olympus |
103 |
-------------------- |
104 |
#!/bin/bash |
105 |
|
106 |
repository_name="${1}" |
107 |
repository_path="${3}" |
108 |
|
109 |
[[ ${repository_name} == "olympus" ]] || exit 0 |
110 |
|
111 |
source /lib/gentoo/functions.sh |
112 |
|
113 |
# Number of jobs for egencache, default is number or processors. |
114 |
parallel_jobs="$(nproc)" |
115 |
|
116 |
# caches |
117 |
ebegin "Fetching pre-generated metadata cache for ${repository_name}" |
118 |
rsync -a |
119 |
rsync://olympus.lan.localdomain/gentoo-portage/metadata/md5-cache/ |
120 |
"${repository_path}"/metadata/md5-cache/ |
121 |
eend $? |
122 |
|
123 |
ebegin "Updating metadata cache for ${repository_name}" |
124 |
egencache --jobs="${parallel_jobs}" --repo="${repository_name}" |
125 |
--update --update-use-local-desc |
126 |
eend $? |
127 |
|
128 |
#DTD's |
129 |
DTDDIR="${repository_path}"/metadata/dtd |
130 |
ebegin "Updating DTDs" |
131 |
if [[ -e ${DTDDIR} ]]; then |
132 |
git -C "${DTDDIR}" pull -q --ff |
133 |
else |
134 |
git clone olympus.lan.localdomain:/usr/portage/metadata/dtd |
135 |
"${DTDDIR}" |
136 |
fi |
137 |
eend "$?" |
138 |
|
139 |
#GLSA's |
140 |
GLSADIR="${repository_path}"/metadata/glsa |
141 |
ebegin "Updating GLSAs" |
142 |
if [[ -e ${GLSADIR} ]]; then |
143 |
git -C "${GLSADIR}" pull -q --ff |
144 |
else |
145 |
git clone olympus.lan.localdomain:/usr/portage/metadata/glsa |
146 |
"${GLSADIR}" |
147 |
fi |
148 |
eend "$?" |
149 |
|
150 |
# herds |
151 |
ebegin "Updating herds.xml" |
152 |
rsync -a |
153 |
rsync://olympus.lan.localdomain/gentoo-portage/metadata/herds.xml |
154 |
"${repository_path}"/metadata/md5-cache/ |
155 |
eend $? |
156 |
|
157 |
#news |
158 |
NEWSDIR="${repository_path}"/metadata/news |
159 |
ebegin "Updating news items" |
160 |
if [[ -e ${NEWSDIR} ]]; then |
161 |
git -C "${NEWSDIR}" pull -q --ff |
162 |
else |
163 |
git clone olympus.lan.localdomain:/usr/portage/metadata/news |
164 |
"${NEWSDIR}" |
165 |
fi |
166 |
eend $? "Try to remove ${NEWSDIR}" |
167 |
-------------------- |
168 |
|
169 |
|
170 |
mv /usr/portage /usr/portage.old |
171 |
|
172 |
git clone olympus.lan.localdomain:/usr/portage /usr/portage |
173 |
|
174 |
echo 'olympus' > /usr/portage/profiles/repo_name |
175 |
|
176 |
echo 'masters = olympus' > /usr/local/portage/metadata/layout.conf |
177 |
|
178 |
echo 'thin-manifests = true' >> /usr/local/portage/metadata/layout.conf |
179 |
|
180 |
cp /usr/portage/metadata/layout.conf |
181 |
/usr/local/portage/metadata/layout.conf |
182 |
|
183 |
cp /etc/portage/repos.conf/olympus.conf /usr/share/portage/config/repos.conf |
184 |
|
185 |
emerge --sync |
186 |
emerge --sync |
187 |
|
188 |
* the first sync will complete the process, the second is to check there are |
189 |
no errors. |