| 1 |
I was tracing into why sources.gentoo.org is so slow at times, and |
| 2 |
gives errors, timeouts etc. I found that there is wget being used |
| 3 |
against the site, with a lot of rapid-fire requests for the *checkout* |
| 4 |
URLs - most often for entire directories. And very often, right when |
| 5 |
wget is being used, the sources.g.o performance sucks badly. |
| 6 |
|
| 7 |
The wget requests account for 25-30% of the daily requests to |
| 8 |
sources.gentoo.org. It's meant for browsing, not pulling. |
| 9 |
|
| 10 |
If you are pulling multiple files, or some file on a regular basis, you |
| 11 |
should be using the anoncvs/anonsvn systems instead. |
| 12 |
|
| 13 |
export ANON=":pserver:anonymous@××××××××××××××.org:/var/cvsroot" |
| 14 |
Listing the files for a package: |
| 15 |
# cvs -z0 -d $ANON rls gentoo-x86/$CAT/$PN/ |
| 16 |
|
| 17 |
Grabbing an entire package: |
| 18 |
# T=`mktemp -d` |
| 19 |
# cvs -z0 -d $ANON co -d $T gentoo-x86/$CAT/$PN |
| 20 |
|
| 21 |
Grabbing a single file, without a temp: |
| 22 |
# cvs -z0 -d $ANON co -p gentoo-x86/$CAT/$PN/$FILENAME >$OUTPUTFILE |
| 23 |
|
| 24 |
Pursuant to the above, the any useragent matching /^Wget/ will be |
| 25 |
blocked from the 'gentoo' and 'gentoo-x86' repos of sources.gentoo.org |
| 26 |
as of July 14th. |
| 27 |
|
| 28 |
Either change to using the proper anonymous service, or change your |
| 29 |
useragent to describe what you are doing with the service, so that I can |
| 30 |
specifically ban your user-agent if it's causing too much load. |
| 31 |
|
| 32 |
-- |
| 33 |
Robin Hugh Johnson |
| 34 |
Gentoo Linux Developer & Infra Guy |
| 35 |
E-Mail : robbat2@g.o |
| 36 |
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 |
Archives