Re: [gentoo-soc] GSoC 2013: Log collector

[Diego Elio Pettenò]

Hi Antanas,

On 29/04/2013 02:48, Antanas Uršulis wrote:
> I've tried to assess what this project would involve, maybe this could
> be a starting point for the proposal. A lot of this describes how the
> current solution works, so changes might be necessary:

Most likely, yes.

> - conceptually the system should have 3 components: a log
> collector&analyser, a storage backend and a frontend

Correct.

> - it would be integrated with portage:
> --- portage would implement a client which can submit logs to the
> collector, possibly providing information why the package failed
> --- this connection between portage and the collector should rely on
> as little as possible (because any packages providing that
> functionality might break)

Also correct.

> --- it should support IPv6 because that's what is used between the
> container on the tinderbox and the box itself (here's a question
> though: any technical reason why IPv6 was used? I admit I didn't look
> into this too deeply)

Yeah the technical reason is actually two fold:

 - the host for the tinderboxes only has one IPv4 address, so it was
either NAT or IPv6; given that the tinderbox runs isolated networking
through proxy to stop packages using network at build time, NAT was not
a great idea; using IPv6 means that I can still jump on the hosts either
from another IPv6-enabled system or, like in my previous and current
office, straight from my IPv6-enabled workstation;
 - by using IPv6, the name of the tinderbox is found simply by doing a
reverse-lookup of the address, as all the tinderboxes have proper records.

> - the collector & analyser:
> --- receives logs over some protocol
> --- should be able to group logs (receive several log files for a
> failing package and keep them together) (this, depending on the
> implementation, might be part of the portage integration)
> --- matches each line against a regexp, we can look into something
> more extensible if required
> --- organises the files by hostname and submits them to the storage backend

Also correct.

> - the storage backend:
> --- I could start with Amazon's AWS and then move to something
> standalone (how much data is there to store, actually? 1/10/100 GB?
> and how large can a single log file become?)

I've seen log files getting over 1GB (yes I know it's crazy) but that's
relatively rare. I don't have a quick assessment of the total storage
over the past year unfortunately.

> --- keeps the logs and also a simple database that would hold
> information about the log groups (package, date, links to log files,
> etc.)

Correct.

> - the frontend:
> --- displays a list of packages that have matches
> --- should be integrated with bugzilla; one can see open bugs for a
> selected package, and also file a new bug
> --- should be password protected

Also correct. Do note that one of the things that the frontend has to do
is being able to _attach_ the data rather than just link to it (which is
what I've been doing myself up to now).

> Comments/additions greatly appreciated. Now, regarding the Gentoo
> application template, I have actually a long time ago submitted a
> one-line workaround patch[1] for openoffice, but that probably doesn't
> qualify. Could you point me (general direction is ok) towards
> something I could fix for my application?

I'm not sure if it makes much sense to sweat fixing something, given
that we're talking about writing a series of webapps and systems. It
might be better for you to point at any kind of similar work you ever done.

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@×××××××××.eu — http://blog.flameeyes.eu/