Re: [gentoo-user] aggregate logs into Elasticsearch

[Stefan G. Weichinger]

Am 03.04.20 um 17:57 schrieb Ralph Seichter:
> * Stefan G. Weichinger:
>
>> My goal:
>>
>> collect logs of postfix, nginx into the docker-containers running ES,
>> Kibana .. and learn my way from there.
>
> If you are not dead-set on Elasticsearch et al, I propose considering
> MongoDB as an alternative.
>
> There are syslog Modules that allow logging into MongoDB directly. On
> the DB side, collections (roughly equivalent to tables in relational
> databases) can be limited by size or by age, meaning that removing older
> data will happen automatically if you so wish.
>
> MongoDB also makes it easy to add data from sources with different data
> makeup to shared collections, because there is no rigid table structure.
>
> For analysis, MongoDB includes its own Aggregation Framework[1], which
> is a very powerful and versatile. While probably not relevant to your
> needs right now, It even comes with built-in geolocation search
>
>   [1] https://docs.mongodb.com/manual/core/aggregation-pipeline/
>
> I think very highly of MongoDB and encourage you to look into it as a
> possibility and as an interesing technical concept.

Thanks for the feedback.

I am not at all set on ES, it just was part of an article I read lately,
and so I started with that docker-compose stack example.

Yesterday I spent quite a while trying to pipe the journald entries into
the fluentd container, quite complicated and messy in a way.

Maybe I look into mongodb as well, for example I found this small howto:

https://www.fluentd.org/guides/recipes/maillog-mongodb

In the end I look for a solution to aggregate (systemd/journald) logs
into one pile of data and to be able to analyze stuff there.

All these solutions seem rather complicated and overly "academic" to me
... but that might be my newbie status in this area.