Gentoo Archives: gentoo-dev

From: "Andreas K. Huettel" <dilfridge@g.o>
To: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] instability
Date: Mon, 05 Dec 2011 11:49:47
Message-Id: 2345971.y9mealLpxW@grenadine
In Reply to: [gentoo-dev] instability by Alec Warner
Seriously, what do we gain from crawlers accessing  I cant 
really remember seeing it once in a google query result... 

Possibly it would not even be required to deny all requests, but just deny 
everything related to ancient history...

> Hello, > > For a while has been puttering along and its health > has slowly declined. We migrated it to some newer shiny hardware in an > attempt to mitigate the problem but that did not pan out. 90% (or > more) of traffic is crawler bots and not actual > humans. That being said; if we cannot serve requests to the bots > within our timeouts we serve 500's instead which is never really what > we want (particularly when we spent 20s of CPU to calculate 80% of the > response only to see the client timeout :/.) > > The majority of the expensive requests are related to package.mask and > use.local.desc queries by crawlers. Like crawling the entire 13000 rev > history for package.mask (or similar.) > > While it is likely we will monkey patch viewvc to be less wasteful; in > the meantime I have removed use.local.desc from > (and also anoncvs, because they share the same repo.) I hope this is a > short term (order of weeks) hack. > > -A
-- Andreas K. Huettel Gentoo Linux developer kde, sci, arm, tex, printing


Subject Author
Re: [gentoo-dev] instability Alec Warner <antarus@g.o>