Re: [gentoo-dev] Gentoo stats server/client @ 2009-06-21

[Sebastian Pipping]

First thanks for sharing your concerns and setup bits.
That's the right thing at the the right time.



Robin H. Johnson wrote:
> Relevant to this, I might not want to disclose my profile inheritance
> tree. Here's one of them for you:
> /etc/make.profile
> /etc/managed-portage/hosts/build_webdb/make.profile
> /etc/managed-portage/common/post/make.profile
> /etc/managed-portage/class/webdb/make.profile
> /etc/managed-portage/class/db/make.profile
> /etc/managed-portage/class/web/make.profile
> /etc/managed-portage/common/pre/make.profile
> /etc/managed-portage/location/surrey/make.profile
> /etc/managed-portage/hwtype/nehalem/make.profile
> /usr/portage/profiles/default/linux/amd64/2008.0

Which of these is the target of the /etc/make.profile link?
The last one?  My current approach resolves the soft link and
cuts of the profiles dir prefix.  So in case it's the last for
you that would be

  default/linux/amd64/2008.0

To auto-filter profiles would parsing profiles.desc work?
Would a synced CVS checkout of
<http://sources.gentoo.org/viewcvs.py/gentoo-x86/profiles/>
give anything more that I could or should use?


>> For Overlays ..
>>     we filter out overlays not located below /usr/local/portage/layman/.
> This is going to be fail.
> 1. That's not the only location used for layman.
> - At home: /code/gentoo/layman/
> - At work: /usr/local/portage-layman/
> - Gentoo Infra: /usr/portage/local/layman/
>
> 2. Just because an overlay is distributed by layman does NOT mean that
>    it's safe to disclose the existence of, within Gentoo infra, we do
>    this in layman.cfg:
> overlays  : http://www.gentoo.org/proj/en/overlays/layman-global.txt
>             file:///etc/layman/infra-overlays.xml

I see.  How about this approach instead:

- Get list of overlays from layman-global.txt, through either

  A) Download and keep a snapshot of layman-global.txt in sync ourselves

  B) Use heuristic on layman's cache

     - Resolve ${cache} from /etc/layman/layman.cfg

     - Parse all ${cache}/cache_*.xml files using the Layman API

     - Compare the list of overlays each file provides against
       a hardcoded snapshot of overlay names ("akoya alexxy arcon ..")

     - Assume the file with the highest count of matches for
       layman-global.txt if the count is >=50 of the number hardcoded
       overlays

- Take the official tree and globa overlays (overlays from
  layman-global.txt) into  account for statistics

  - Resolve ${storage} from /etc/layman/layman.cfg

  - Include ebuilds from ${storage}/{global,overlays,here} and
    /usr/portage/

What it does not catch is people putting their own ebuilds
right into the main tree.  As they lose them all on the next sync
are we safe to assume that no one really does that?
If not are there alternatives to comparing to a synced checkout
of gentoo-x86 (either rsync or CVS)?

Any concerns or ideas for improvement?


> While I don't mind disclosing the list of overlays we have in infra,
> other large-scale use of layman might not be happy to disclose it.

Agreed.


> 3. For one of my work overlays, we have a custom category called
>    'ih-int', for our internal ebuilds (some just meta ebuild, others
>    full applications). I might not want to disclose just those package names.

Right.  With the approach described above the whole overlay is ignored.



Sebastian