[gentoo-amd64] Re: Systemd migration: opinion and questions

[Duncan]

Rich Freeman posted on Tue, 24 Feb 2015 16:44:59 -0500 as excerpted:

> On Tue, Feb 24, 2015 at 3:15 PM, Marc Joliet <marcec@×××.de> wrote:
>>
>> == Things I have *not* gotten rid of (yet) ==
>>
>> I plan on uninstalling syslog-ng, but haven't done so yet.  I simply
>> feel better waiting a bit, even though I don't run it anymore.  Man, I
>> feel silly after typing that...
>
> Yeah, took me a while to uninstall it, but I don't find much value in
> running both.

First, welcome to systemd! =:^)  I may respond to other parts too but 
this one's reasonably easy, while my choice is different, so I'll explain 
what and why...

Intro: I chose to keep syslog-ng running here, while I (of course) have 
journald running as well.  I believe the way I have configured both plays 
to the advantages of each, while discounting the weak areas of each as 
the other one covers that area.

What:

1) Journald is configured to use volatile storage (the /run/log/journal 
subdir, with /run being tmpfs) ONLY, effectively limiting it to single-
session.

2) Syslog-ng is configured pretty much as before, except that I have its 
systemd USE flag turned on, which builds it with journald support so the 
two cooperate a bit better.

Without looking too much into details, I believe USE=systemd for syslog-ng 
tells it to build against a library systemd/journald provides for that 
purpose, such that syslog-ng's "system" source listens to journald 
instead of /dev/log and the kernel (dmesg) directly.

What I can say is that while I've seen complaints from others about 
loggers getting duplicate entries (from systemd and the kernel both, for 
instance) in some cases, or missing entries in others (the OP, here), 
I've not noticed anything like that, here, and I believe it's due to the 
above -- only having the "system" and internal syslog-ng sources 
configured so syslog-ng isn't getting the same message from two sources, 
and building with the systemd USE flag, which basically configures syslog-
ng's system source to be lazy and let journald feed it, instead of doing 
all the work of fetching and parsing the kernel and syslog socket on its 
own.

Why:

I love the journald/systemd integration -- having the last few log 
messages on systemctl status <service> makes troubleshooting within the 
same session **MUCH** easier, and the other information journald keeps is 
also useful.  But, for me at least, that's most useful in the same 
session.  Once I reboot and the information I'm looking for is in a 
previous session, the usefulness of that extra information goes down 
quite a bit.

Meanwhile, for logs beyond one session, I still prefer traditional text-
based logs, tho I suppose that's partly because that's what I'm most used 
to.  But there's several advantages to it that I see:

a) If the system crashes, partially corrupted in-the-crash text logs can 
be of at least some use after a reboot.  Binary journals, not so much.

b) As far as I could find, journald has absolutely no mechanism to filter 
incoming (pre-storage) log entries, while syslog-ng has had (pre-storage) 
filter mechanisms for ages.  Journald seems to have all sorts of outgoing 
post-storage filter-on-output options, some of which are pretty nifty, 
but there have been several times in the past where logs filled up with 
"noise", thousands of repeat entries for something or other that occurs 
regularly, say once a minute, that I simply do not care about and don't 
want in my logs at all, but which I can't directly stop at the source for 
whatever reason.  One of them was a kernel bug that triggered for a few 
kernel cycles, then eventually went away, another was a ksysguard 
triggered message, with that ksysguard graph updating every second, etc.  
Unfortunately, while journald makes it easy to filter these out on 
output, I found no way of preventing their appearance in the journal at 
all.

But syslog-ng lets me dump them without ever actually logging them, or 
route them to a different log file if I prefer, keeping my primary logs 
clean. =:^)

As a counter-point, however, at least journald does compression to some 
degree, so tens of thousands of identical or nearly identical messages 
don't take up as much room as you might expect as they compress very 
well. =:^)  I guess after the first instance, dups basically consist of a 
timestamp and a pointer to the first message.  So they don't take much 
space.  Which is fortunate when some "noise" message starts appearing 
once a second... and you're storing it in tmpfs, thus in memory!  
Obviously persistent logs would be similarly space efficient, but I still 
don't like the idea of not being able to filter out the noise, BEFORE it 
hits permanent storage, so I just make sure it /doesn't/ hit permanent 
storage. =:^)

c) I use btrfs for my primary filesystems, and btrfs and journald's 
binary-format journals don't play so well together.  FWIW, the latest 
systemd-219 has lots of improvements for btrfs users, including some new 
features that are btrfs-specific (which I never-the-less don't expect to 
use right away as they're mostly VM/container oriented and I don't do 
much of that sort of thing, but I may well eventually use them), as well 
as some changes that should make journaling on btrfs work somewhat 
better.  So this point should be to some degree eliminated after that 
stuff stabilizes tho part of the fix is journald setting nocow on the 
journal files and that has other implications I'll skip further 
discussion of here.  However, by limiting journald to tmpfs only, I still 
get the best benefits of systemd/journald integration (like status 
including the last few log messages as mentioned above), while 
eliminating the entire class of btrfs/journald issues.  Best of both 
worlds. =:^)

d) Tho I don't do anything with it in my syslog-ng config here, syslog-ng 
can actually make use of some of the extra information journald passes to 
it, including trusted per-app filtering and routing.  One of the problems 
that the old syslog approach had was that anything that could log 
messages could claim to be anything it wanted, and syslog really couldn't 
tell whether it was telling the truth or not, all it could do is sort and 
report the messages based on what was claimed.  A big bullet-point 
feature of systemd/journald is that because systemd enforces service 
separation via control-classes and the like, there's a much stronger 
guarantee that what systemd/journald says is service X, is REALLY service 
X and not service Y!  But that's not limited to journald.  Syslog-ng has 
for several versions now had the ability to filter, route and report on 
out-of-log-channel information such as the extra properties that journald 
associates with each message, that make features such as systemctl status 
reporting the last few log messages from that service possible, because 
systemd/journald actually tracks it, AND passes it on to other loggers as 
well now, providing they're built to use the API provided by the systemd 
library I mentioned above.  And again, for gentooers, syslog-ng is built 
with that API, against that library, based on USE=systemd.  So once 
that's there, if desired, you can do all sorts of fancy filtering/log-
routing/etc based on information such as the application/service name, 
that systemd passes along to syslog-ng.

And again, with syslog-ng, this can be done BEFORE the actual logging if 
desired, instead of logging everything and filtering only on output, as 
journald does. =:^)

e) I'm simply more comfortable dealing with text-based logs, since I'm 
used to them and can use whatever tools I want on them, as well as being 
able to split/sort/route to multiple log files in as simple or complex a 
setup as I want. =:^)

Summary:

For me, splitting up logging duties so journald does the collection, but 
only does session journaling to tmpfs, while syslog-ng still handles the 
"persistent" logging, gives me the best of both worlds.  I love systemd/
journald features such as log-messages-in-status and having seen how 
useful it is, would seriously find it hard to do without.  But at the 
same time, I can't bring myself to accept persistent logging without the 
ability to filter out the noise before it hits persistent storage, and 
journal storage on btrfs is problematic anyway, with both problems nicely 
solved by handing off to syslog-ng for my persistent storage needs while 
keeping journald for same-session use, so that's exactly what I do. =:^)


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman