Re: [gentoo-soc] Re: Gentoo stats server/client,

[Joachim Bartosik]

This idea looks interesting so if you don't mind I'll join the thread.
I tried to cut everything short but it looks too long anyway :/ And trying
too keep it short probably made some parts understandable so please ask.
If you see * scroll down to end of email for explanations.


>> There have been many stats projects in the past that have failed due to
> >> various reasons. A simple question is: How are you planning on making
> >> your idea/proposal not fail? ;)
>

By being lazy and putting as much work on others as possible.

Authentication/ security overview

The idea from 2006 ( to create account one has to ask for id and  submit
some data) makes usage very simple for users ( they don't even need to know
anything about authorisation), but unluckily it's very easy to write
"client" that would  submit a lot data that would spoil the data ( I guess
that's a major issue with authentication and security).

To solve this problem I'd use less comfortable for users solution: user wold
have to create an account using an email ( of course it wouldn't be stored,
I'd store some one-way injective function of it*) and click an emailed link.
There would be no need for password - to confirm his[her] actions [s]he
would just click an emailed link.

Each user ( email) would have a hosts** limit ( probably set in server
configuration) 2 or 3 by default ( enough for average user, not enough to
easily spoil data). After some time of inactivity host/ account would be
removed.

The problem starts if one would need to get more hosts per account, right
now I have some ideas ( none very good):
- the easiest to implement method is "please email our admin and explain why
do you need them" but it's user unfriendly and admin unfriendly.
- give really big limit on the hosts per email - it would be easy to inject
a lot of false data, but it's easier to remove then in 2006 auth( identify
wrongdoing emails and delete their hosts).
- require users to give some non-free ( free as in beer) email to reduce
possibility of using fake emails and give big hosts limit.

 I'd try to keep need to click email's to minimum - registration and
administrative tasks ( like removing hosts from account).



Components:

Client

Probably in python to take advantage of all the work portage developers have
done and save me work. I'd be a simple run-me-from command line ( cron)
program sending arch, all installed cpv and their USE ( for sure and before
end of summer) and maybe some more if time allows ( "A daemon with 2 working
modules is better than a daemon with 10 half finished ones.").  Maybe [if
time allows] GUI wrapper to run it in tray.

Server:
Would be split into several independent programs ( to save me work). All
except first one would be written in python.

User communication:
Thanks for Rest idea - i thought about using HTML/ HTTPS but making it's
stateless saves a lot of work. To save me some work I'd start with Apache +
php + MySQL, one path per action ( register host, register user, send data,
...). It'd put received data in MySQL ( not verify if their correct, simply
get data, and put it in table with information who and when sent it). It's
not a very elegant solution ( and may turn out to be slow) so -if the time
allows and there is need to- I'll rewrite it in python.

Data gathering:
It'd take data provided by user communication module, decompress it, apply
deltas etc. to create all-the-information-available about current state of
hosts.

Cleaner:
Run from time to time ( by cron, frequency adjusted to needs). Remove hosts
and users that do not send data ( to conserve space) etc.

Achiever:
Run from time to time ( cron, as needed). Data gathering provides only
information about hosts *right now*. Achiever would generate statistics (
like package popularity ( % hosts that installed it)) and store them to make
historical data available ( storing all host states history would be
extremely excessive).

* The one-way part means that there is no easy way to get users emails even
if someone gets access to the all data stored on server. The injective part
means that no two emails will generate the same output, so no two users will
get the same account. Hashes won't work because they are not injective
functions but I'm almost sure someone already wrote functions like that. I
don't recall any right now, but I'll have plenty of time to look for them or
in worst case write one me self ( easy: create asymmetric pair of keys,
throw private to /dev/null so none can decrypt it and encrypt emails with
public one).

** 1 host == 1 data set ( installed packages, arch etc.)



I realized I forgot to tell who am I:

I'm Joachim live in Poland ( UTC + 1). Study mathematics ( 3rd year). Use
Gentoo since 2005 ( as main desktop OS) or 2004 ( first contact). Code since
2003 ( training for/ participating in http://www.oi.edu.pl English version
available - look in the right top corner) or 2001 ( started to play with
Vbasic ).  Cannot drink black tea. Right now extremely tired after sleepless
weekend ( due to several breakdowns at home).
Good night.