Gentoo Logo
Gentoo Spaceship

Note: Due to technical difficulties, the Archives are currently not up to date. GMANE provides an alternative service for most mailing lists.
c.f. bug 424647
List Archive: gentoo-dev
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
To: gentoo-dev@g.o
From: Alec Warner <antarus@g.o>
Subject: Re: instability
Date: Mon, 5 Dec 2011 08:27:00 -0800
On Mon, Dec 5, 2011 at 3:48 AM, Andreas K. Huettel <dilfridge@g.o> wrote:
> Seriously, what do we gain from crawlers accessing  I cant
> really remember seeing it once in a google query result...

We want the site searchable.

> Possibly it would not even be required to deny all requests, but just deny
> everything related to ancient history...
>> Hello,
>> For a while has been puttering along and its health
>> has slowly declined. We migrated it to some newer shiny hardware in an
>> attempt to mitigate the problem but that did not pan out. 90% (or
>> more) of traffic is crawler bots and not actual
>> humans. That being said; if we cannot serve requests to the bots
>> within our timeouts we serve 500's instead which is never really what
>> we want (particularly when we spent 20s of CPU to calculate 80% of the
>> response only to see the client timeout :/.)
>> The majority of the expensive requests are related to package.mask and
>> use.local.desc queries by crawlers. Like crawling the entire 13000 rev
>> history for package.mask (or similar.)
>> While it is likely we will monkey patch viewvc to be less wasteful; in
>> the meantime I have removed use.local.desc from
>> (and also anoncvs, because they share the same repo.) I hope this is a
>> short term (order of weeks) hack.
>> -A
> --
> Andreas K. Huettel
> Gentoo Linux developer
> kde, sci, arm, tex, printing

Re: instability
-- Chí-Thanh Christopher Nguyễn
References: instability
-- Alec Warner
Re: instability
-- Andreas K. Huettel
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Previous by thread:
Re: instability
Next by thread:
Re: instability
Previous by date:
Re: We need *you* for a USE="selinux" dependency
Next by date:
Re: We need *you* for a USE="selinux" dependency

Updated Jun 29, 2012

Summary: Archive of the gentoo-dev mailing list.

Donate to support our development efforts.

Copyright 2001-2013 Gentoo Foundation, Inc. Questions, Comments? Contact us.