Gentoo Logo
Gentoo Spaceship




Note: Due to technical difficulties, the Archives are currently not up to date. GMANE provides an alternative service for most mailing lists.
c.f. bug 424647
List Archive: gentoo-dev
Navigation:
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Headers:
To: gentoo-dev@g.o
From: Alec Warner <antarus@g.o>
Subject: Re: sources.gentoo.org instability
Date: Thu, 8 Dec 2011 21:30:51 -0800
2011/12/5 Chí-Thanh Christopher Nguyễn <chithanh@g.o>:
> Alec Warner schrieb:
>>> Seriously, what do we gain from crawlers accessing sources.gentoo.org?  I cant
>>> really remember seeing it once in a google query result...
>>
>> We want the site searchable.
>
>>>> The majority of the expensive requests are related to package.mask and
>>>> use.local.desc queries by crawlers. Like crawling the entire 13000 rev
>>>> history for package.mask (or similar.)
>
> Would it be feasible to use mod_rewrite to direct the most expensive
> requests to a static copy, which is re-generated every
> ${REASONABLE_TIMEFRAME}?

For now user-agents that look like a bot get sent to
sources2.gentoo.org (via HTTP-302, not a perm redirect) and humans are
good on sources.gentoo.org. Assuming the crawlers and indexing systems
follow the spec; hopefully all our search resutls do not get rewritten
to sources2.gentoo.org (that would surprise me greatly...wait no it
wouldn't ;p)

Robin added a caching layer for some segments of the application; I am
looking at cprofile dumps and discussing pain points with upstream.

-A

>
>
> Best regards,
> Chí-Thanh Christopher Nguyễn
>


References:
sources.gentoo.org instability
-- Alec Warner
Re: sources.gentoo.org instability
-- Andreas K. Huettel
Re: sources.gentoo.org instability
-- Alec Warner
Re: sources.gentoo.org instability
-- Chí-Thanh Christopher Nguyễn
Navigation:
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Previous by thread:
Re: sources.gentoo.org instability
Next by thread:
Re: News item for KDEPIM-4.7 stabilization
Previous by date:
Re: multiple inclusion protection with eclasses
Next by date:
Re: multiple inclusion protection with eclasses


Updated Jun 29, 2012

Summary: Archive of the gentoo-dev mailing list.

Donate to support our development efforts.

Copyright 2001-2013 Gentoo Foundation, Inc. Questions, Comments? Contact us.