Re: [gentoo-soc] Package statistics, progress report #1

[Brian Dolbec]

On Thu, May 19, 2011 at 10:52 AM, Vikraman <vikraman.choudhury@×××××.com>wrote:

>
> Hello everyone,
>
> I'm working on the 'Package Statistics' project [1] for GSoC this summer.
> This is my first progress report.
>
>
> A short summary of my progress:
> -------------------------------
>
> * Created project repository [2] on git.overlays.gentoo.org
> * Read up on RESTful Web Services (from the O'reilly book)
> * Tried to improve my python coding style using the google guide
>  (suggested by Alec)
> * Wrote a simple client in python to collect a few environment variables
>  from portage, list of installed packages with useflags, encode the
>  data in JSON, with proper authentication, and issue a POST to the
>  server
>


Depending how much detail you want to collect, the enalyze modules in
gentoolkit have USE flag filtering to remove the normally hidden flags that
users never see.

(note: enalyze is the new "e" name for the analyse modules)


* Wrote a simple webapp using web.py to handle requests from the above
>  client and save the data to MySQL tables
> * Wrote some documentation to deploy the webapp and client
>
> Issues encountered:
> -------------------
>
> * Choice of portage api vs gentoolkit api: The gentoolkit api is very
>  easy to use, but quite slow [3] compared to the portage api. Alec asked me
>  to use both of them as necessary, but provide an easy way to swap out
>  one in favor of the other at a later time.
>


Yes, that part of gentoolkit is slower than using portage directly as it
does other things than just return the installed cpv list.

1) it has the ability to filter the list using a caller supplied function.

2) it is a generator and doesn't perform the actual read/load immediately,
which saves time for areas of code that may not end up using it, but is
loaded.

That is one reason I did not use it initially in the enalyze modules, but
added support to optionally use it (at someone else's insistence) and the
gentoolkit package object.

Most of what information you will want for stats gathering should be taken
from the enalyze modules. They are specific to the installed pkg db, are as
a result will tend to be faster than the other gentoolkit functions.  If you
look at them closer you will see that they accept a pre-loaded cpv list
(using the same portage-api)  for running different reports in succession.
 As I informed you during the proposal period, the enalyze modules are
nearly tailor made for the information gathering your stats project will be
needing.

If there is some data functions missing from it, I can probably add them in
for you, since I will be adding more reports and functionality to enalyze.

Another thing to keep in mind is that if the code used to gather the
information and report it, is part of gentoolkit.  It is far more likely to
be accepted and used to report to the server.  While I don't know the
numbers (something a successful result of this project would tell) I believe
the majority of gentoo users have it installed.  I believe that all the
previous stats projects have not succeeded fully is that there was an
additional pkg to install that most people did not even know about.


Also I am working on the portage public_api, so if there is some special
functionality that you need from portage, if it is suitable for that api, I
can add it in.




>
> * Get updates from the community on what data should be collected from
>  hosts
> * Try to add more fields to the client/server and modify the SQL tables
>  accordingly
> * Learn more about the portage api and discuss them on #gentoo-portage
>
> My semester exams are currently in progress, and they'll last till the
> end of May. So, I'll not be able to work during these 2 weeks. I'm
> looking forward to get back on 1st June, and continue with my project.
>
>
> --
> Vikraman
>


Brian Dolbec