Gentoo Archives: gentoo-dev

From: Alec Warner <antarus@g.o>
To: Gentoo Dev <gentoo-dev@l.g.o>
Subject: [gentoo-dev] sources.gentoo.org instability
Date: Mon, 05 Dec 2011 07:11:17
Message-Id: CAAr7Pr-XmKGg8byabXqHN=X+PkogLVgLTfMkWLr0tQLCS086Ww@mail.gmail.com
1 Hello,
2
3 For a while sources.gentoo.org has been puttering along and its health
4 has slowly declined. We migrated it to some newer shiny hardware in an
5 attempt to mitigate the problem but that did not pan out. 90% (or
6 more) of sources.gentoo.org traffic is crawler bots and not actual
7 humans. That being said; if we cannot serve requests to the bots
8 within our timeouts we serve 500's instead which is never really what
9 we want (particularly when we spent 20s of CPU to calculate 80% of the
10 response only to see the client timeout :/.)
11
12 The majority of the expensive requests are related to package.mask and
13 use.local.desc queries by crawlers. Like crawling the entire 13000 rev
14 history for package.mask (or similar.)
15
16 While it is likely we will monkey patch viewvc to be less wasteful; in
17 the meantime I have removed use.local.desc from sources.gentoo.org
18 (and also anoncvs, because they share the same repo.) I hope this is a
19 short term (order of weeks) hack.
20
21 -A

Replies

Subject Author
Re: [gentoo-dev] sources.gentoo.org instability "Andreas K. Huettel" <dilfridge@g.o>