Re: [gentoo-soc] GSoC 2013: Log collector

[Antanas Uršulis]

Thanks for the input, I'll write up a draft application and mail it
here later on. In the mean time, I haven't done directly related work,
but I could point to my bachelor's project [1]. It's very much a work
in progress, but the work I did in Cambridge over last summer is not
public yet (really tiny bits of it are here [2]).

[1] https://github.com/aursulis/ciel/tree/shm_blockstore
[2] https://github.com/awm22/NetFPGA-P33/pull/1

On Mon, Apr 29, 2013 at 8:28 AM, Diego Elio Pettenò
<flameeyes@×××××××××.eu> wrote:
> Hi Antanas,
>
> On 29/04/2013 02:48, Antanas Uršulis wrote:
>> I've tried to assess what this project would involve, maybe this could
>> be a starting point for the proposal. A lot of this describes how the
>> current solution works, so changes might be necessary:
>
> Most likely, yes.
>
>> - conceptually the system should have 3 components: a log
>> collector&analyser, a storage backend and a frontend
>
> Correct.
>
>> - it would be integrated with portage:
>> --- portage would implement a client which can submit logs to the
>> collector, possibly providing information why the package failed
>> --- this connection between portage and the collector should rely on
>> as little as possible (because any packages providing that
>> functionality might break)
>
> Also correct.
>
>> --- it should support IPv6 because that's what is used between the
>> container on the tinderbox and the box itself (here's a question
>> though: any technical reason why IPv6 was used? I admit I didn't look
>> into this too deeply)
>
> Yeah the technical reason is actually two fold:
>
>  - the host for the tinderboxes only has one IPv4 address, so it was
> either NAT or IPv6; given that the tinderbox runs isolated networking
> through proxy to stop packages using network at build time, NAT was not
> a great idea; using IPv6 means that I can still jump on the hosts either
> from another IPv6-enabled system or, like in my previous and current
> office, straight from my IPv6-enabled workstation;
>  - by using IPv6, the name of the tinderbox is found simply by doing a
> reverse-lookup of the address, as all the tinderboxes have proper records.
>
>> - the collector & analyser:
>> --- receives logs over some protocol
>> --- should be able to group logs (receive several log files for a
>> failing package and keep them together) (this, depending on the
>> implementation, might be part of the portage integration)
>> --- matches each line against a regexp, we can look into something
>> more extensible if required
>> --- organises the files by hostname and submits them to the storage backend
>
> Also correct.
>
>> - the storage backend:
>> --- I could start with Amazon's AWS and then move to something
>> standalone (how much data is there to store, actually? 1/10/100 GB?
>> and how large can a single log file become?)
>
> I've seen log files getting over 1GB (yes I know it's crazy) but that's
> relatively rare. I don't have a quick assessment of the total storage
> over the past year unfortunately.
>
>> --- keeps the logs and also a simple database that would hold
>> information about the log groups (package, date, links to log files,
>> etc.)
>
> Correct.
>
>> - the frontend:
>> --- displays a list of packages that have matches
>> --- should be integrated with bugzilla; one can see open bugs for a
>> selected package, and also file a new bug
>> --- should be password protected
>
> Also correct. Do note that one of the things that the frontend has to do
> is being able to _attach_ the data rather than just link to it (which is
> what I've been doing myself up to now).
>
>> Comments/additions greatly appreciated. Now, regarding the Gentoo
>> application template, I have actually a long time ago submitted a
>> one-line workaround patch[1] for openoffice, but that probably doesn't
>> qualify. Could you point me (general direction is ok) towards
>> something I could fix for my application?
>
> I'm not sure if it makes much sense to sweat fixing something, given
> that we're talking about writing a series of webapps and systems. It
> might be better for you to point at any kind of similar work you ever done.
>
> --
> Diego Elio Pettenò — Flameeyes
> flameeyes@×××××××××.eu — http://blog.flameeyes.eu/
>