Re: [gentoo-soc] Package statistics reporting tool

[Yanbo Wang]

Thanks for your reply.

2012/3/17 Rafael Goncalves Martins <rafaelmartins@g.o>:
> Hi,
>
> On Fri, Mar 16, 2012 at 12:59 PM, Corentin Chary
> <corentin.chary@×××××.com> wrote:
>> Hi,
>>
>> Ccing Rafael.
>>
>> On Thu, Mar 15, 2012 at 2:44 PM, Yanbo Wang <wybb86@×××××.com> wrote:
>>> Hi everyone,
>>>
>>> I'm a post graduate student from China. I'm interested in the idea about
>>> Package statistics reporting tool and I'd like to take part in this year's GSoc
>>> as a student.
>>>
>>> I have read the code written by Vikraman and deployed the server on my gentoo.
>>> The server gathered a lot of information about environment and
>>> packages installed.
>>> I could get the statistic through web interface or terminal.That's
>>> very great, I really want
>>> to contribute to the project.
>>> Please tell me this year's focus, and I have some ideas to share.
>>> 1. Upload more information such as developers' email, whether the packages
>>> installed is stable or not,  the crash info.
>>> 2. Some statistics like popularity(recently installed packages, least
>>> popular packages)
>>> or the relationship between packages could be done in a cronjob.
>>> 3. Integrate with bugzilla. Developers can easily get the bugs of
>>> packages through
>>> statistic web interface or terminal, like PyBugz [1]. These will help
>>> developers find bugs
>>> of their packages easier.
>
> @Yanbo, thanks for your interest. Some of your ideas are great, but
> one of the things that we should keep in mind for this project is
> privacy. We should make sure that the user is aware of what data he is
> sending for our servers, and disable it if needed. The code should be
> able to handle this properly. I think that we shouldn't install a
> cronjob automatically, for example.
>

Of course, privacy is an important part of this project. Currently,
users can decide what data they want to sending  by payload.cfg file
and it will be easily extended if we add more features. The client
generates a UUID for the host on installation and the server uses UUID
to identify the host and packages installed.
Maybe my expression was not clear, a cronjob could be installed on the
server, not on the host, because the extraction of some interesting
statistics, like relationship between packages, will be time-consuming
and we should preprocess them with a cronjob. These statistics may be
used for recommending related packages or finding bugs between
packages.

>>> About me:
>>> I’ve had 3 years’ experience in java development (Java Web and Swing), 2 years'
>>> in python development (Django) and also be fimilar with c/c++.  I’ve been using
>>> gentoo for over 1 year and I really want to contribute a little to
>>> gentoo community.
>>>
>>> [1] http://www.liquidx.net/pybugz/
>>
>> I took a quick look at last year code, and I think the project would
>> be easier to extend if it was using django. This would avoid writing
>> custom SQL queries, etc... And you can probably steal some code from
>> euscan to do that.
>> Note that I'm not the mentor for this project and it is just an idea.
>
> @Corentin: During last year's gsoc I talked quickly with Vikraman and
> Alec about the web framework being used. I agree that web.py isn't the
> better tool for the job, I think I suggested flask that time, don't
> recall correctly.
>

Django is okey for me and I can learn something related from euscan.
Thanks for Corentin's advice.

> This topic should be well discussed, because any change of framework
> will spend a considerable amount of time, considering that we just
> have 3 months.
>
>> I think this years goals should be:
>> - stabilization of the client, and integration with the base system,
>> documentation during installation, etc...
>> - adding more features to the web interface, like what we can find  on
>> http://popcon.debian.org/ (charts, sortable tables, overlays usage,
>> etc..)
>> - integration with other gentoo webservices (how ?)

Like PyBugz [1], developers can get bugs through terminal when they
query packages, rather than Bugzilla website. In addition, we can add
bugs' links to Bugzilla website in packages statistic page.

>> - putting the project in production of gentoo-infra
>>
>
> @Corentin: I tend to agree with these goals. Thanks for the answer.
>
> @Yanbo, your ideas are good, but keep security and privacy in mind. I
> think that these are the key points of the success of this project.
> Also, automatic bug reports are cool, but we need to stabilize the
> basic features first.
>

Yes, thanks for yours and Corentin's suggestion, stabilization and
adding more features to the web interface are most import to complete.
There has been one package named abrt [2], and this tool helps users
easily report bugs of packages to Bugzilla. My idea about bug report
is reporting package name with some comment to the server with
terminal like current gentoostats-send script when the package crashs.
On the server side, reports are analysed offline. We can extract the
usefull things and help developers find bugs. For example, most of
users build one package with the same CFLAG have the same problem or
install one package with another package together have the same crash
info.

> Best regards,
>
> --
> Rafael Goncalves Martins
> Gentoo Linux developer
> http://rafaelmartins.eng.br/
>

[1] http://packages.gentoo.org/package/www-client/pybugz
[2] http://packages.gentoo.org/package/app-admin/abrt

-- 
Thanks,
Yanbo Wang