Re: [gentoo-user] aggregate logs into Elasticsearch

[Stefan G. Weichinger]

Am 04.04.20 um 16:02 schrieb Ralph Seichter:
> * Stefan G. Weichinger:
>
>> Maybe I look into mongodb as well, for example I found this small
>> howto: https://www.fluentd.org/guides/recipes/maillog-mongodb
>
> That looks unnecessarily complicated to me. While you can of course move
> data from an existing log file into MongoDB, I find configuring syslog
> to use a MongoDB destination (in addition to your files or as a full
> replacement) much easier.
>
> See [1] section "Storing messages in a MongoDB database". I have also
> done it with rsyslog, but that took a bit more work.
>
> Here's a syslog-ng destination I use. Note that using uri() allows
> passing parameters to modern MongoDB drivers which the older servers()
> statement cannot cope with.
>
>   destination d_mongo {
>     mongodb(
>       uri("mongodb://user:pw@hostname:27017/syslog?authSource=admin&ssl=true")
>       collection("messages")
>       value-pairs(
>         scope("selected-macros" "nv-pairs")
>         pair("DATE", datetime("$UNIXTIME"))
>         pair("PID", int64("$PID"))
>         pair("SEQNUM", int64("$SEQNUM"))
>         exclude("HOST*")
>         exclude("LEGACY*")
>         exclude("SOURCE*")
>         exclude("TAGS")
>       )
>     );
>   };
>
> Values are strings to begin with. This example excludes some values I am
> not interested in, and performs type conversion on others, for example
> mapping DATE to MongoDB's date/time data type (see ISODate) and PID to a
> numeric value. Conversion can of course happen during analysis, but
> since syslog-ng is smart enough to do it when writing data, I prefer
> that.
>
> [1] https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/37#TOPIC-956524

Thanks a lot ... I think I will postpone that project ;-)