Re: [gentoo-user] syslog-ng filtering

[Alan McKinnon]

On Wednesday 17 March 2010 22:16:20 Ralph Slooten wrote:
> Fantastic, you hit the nail right on the head! Works like a charm now.
>
> Now I'm wondering how it is you found out that it was this way and not the
> other? Robert maintains the documentation for rsync which I did look at,
> but with 225 pages I wasn't able to find this useful piece of information.
> Man syslog-ng.conf does not explain it either, in fact I searched Google
> and found several "tutorials", none mentioning this ;-)

I read documentation, man pages and google all day every day, some things just 
get intuitive :-)

Seriously though, there are a few hints. Syslog-ng's config file format was 
written by programmers for programmers to be understood by programmers. That 
may not have been the stated intent, but it is how things turned out. The 
syntax is exactly that of C, all the way down to braces and statement 
terminators. So, when reading the docs, I flicked the switch that puts my 
brain in C-mode.

Also, there's an example in the admin guide pdf chapter 3 "Configuring syslog-
ng", something like:

match("string" value(MESSAGE);

It says that MESSAGE is exactly that and must not be dereferenced with "$"

That was a dead give-away

>
> Maybe I'm the idiot here, however I thought that this was a common way of
> getting rid of unwanted crud from the syslog?

It IS the ideal way to pre-filter logs based on the message content. Pre 
version 3, you could only match on the entire message, so the feature to be 
able to search just a user-defined chunk of the log entry is a major plus
 
> Also, I just read the gentoo-wiki site page again and it says :
>
> filter f_shorewall { not match("regex" value("Shorewall")); };  #
> Filter everything except regex keyword Shorewall
>
> Surely this is the exact same mistake I made? Either that or I'm reading it
> wrong....

No, you are not reading it wrong - the gentoo guide is wrong. It's a common 
mistake, as the syntax looks like it's a name-value pair. To my mind, the 
label "value" should instead be "field" or some synonym of that.

All the evidence indicates to me that the syntax makes sense once you "get" 
how it works, but most folks' initial assumption about it is wrong, and the 
developer never spotted his serious case of being blinded by his own 
understanding.

I see Robert responded here earlier. Perhaps he'll see this post and re-look 
at that section in a new light with a view to making a patch



>
> On 17 March 2010 23:39, Alan McKinnon <alan.mckinnon@×××××.com> wrote:
> > On Wednesday 17 March 2010 01:22:59 Ralph Slooten wrote:
> > > Hi all,
> > >
> > > Has anyone here worked out how to filter out syslog messages using
> > > syslog-ng v3? The old syntax doesn't work (well complains bitterly
> > > about performance and says to use regex), and no matter what I try I
> > > cannot get the new syntax to work :-/ I have a syslog-ng server which
> > > logs to MySQL for multiple clients in a network, however the database
> > > just keeps
> >
> > growing
> >
> > > with irrelevant data I'd prefer to just quietly ignore on the server
> >
> > side.
> >
> > > I'm trying to filter out (exclude) messages such as:
> > >   (root) CMD (/root/bin/vmware-checker)
> > >
> > > and
> > >
> > >   (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
> > >
> > > ==============
> > > filter myfilter {
> > >
> > >         not match("regex" value("\/usr\/sbin\/run-crons"))
> > >         and not match("regex" value("vmware-checker"));
> > >
> > > }
> >
> > Hah! this caught me out too.
> >
> > The value of "value" cannot be anything arbitrary - syslog-ng has no clue
> > what
> > you mean. The value is a field name, either a pre-defined one, or
> > something you defined using a parser. The docs are ambiguous on this,
> > it's not clear that the supplied values are abstracts. You are truing to
> > search for the string "regex" in a field called /usr/bin/vmware-checker.
> >
> > Which obviously will not work.
> >
> > I think you want:
> >
> > match("\/usr\/sbin\/run-crons" value "MESSAGE")
> >
> > Note that it is MESSAGE. You want the field name, not it's dereferenced
> > value.
> >
> > > log {
> > >
> > >         source(src);
> > >         source(remote);
> > >         filter(myfilter);
> > >         destination(d_mysql);
> > >
> > > };
> > > ===============
> > >
> > > However they just keep coming through the filter (ie: not matching the
> >
> > "not
> >
> > > match" filter). I've tried escaping the slashes, not escaping them ...
> >
> > even
> >
> > > partial words, but I obviously am missing something somewhere.
> > >
> > > Anyone have any ideas?
> > >
> > > Thanks in advance,
> > > Ralph
> >
> > --
> > alan dot mckinnon at gmail dot com

-- 
alan dot mckinnon at gmail dot com