Re: [gentoo-amd64] Re: Systemd migration: opinion and questions

[Marc Joliet]

Am Wed, 25 Feb 2015 10:13:14 +0000 (UTC)
schrieb Duncan <1i5t5.duncan@×××.net>:

> Rich Freeman posted on Tue, 24 Feb 2015 16:44:59 -0500 as excerpted:
>
> > On Tue, Feb 24, 2015 at 3:15 PM, Marc Joliet <marcec@×××.de> wrote:
> >>
> >> == Things I have *not* gotten rid of (yet) ==
> >>
> >> I plan on uninstalling syslog-ng, but haven't done so yet.  I simply
> >> feel better waiting a bit, even though I don't run it anymore.  Man, I
> >> feel silly after typing that...
> >
> > Yeah, took me a while to uninstall it, but I don't find much value in
> > running both.
>
> First, welcome to systemd! =:^)  I may respond to other parts too but
> this one's reasonably easy, while my choice is different, so I'll explain
> what and why...
>
> Intro: I chose to keep syslog-ng running here, while I (of course) have
> journald running as well.  I believe the way I have configured both plays
> to the advantages of each, while discounting the weak areas of each as
> the other one covers that area.
>
> What:
>
> 1) Journald is configured to use volatile storage (the /run/log/journal
> subdir, with /run being tmpfs) ONLY, effectively limiting it to single-
> session.

Right, I remember that from your thread a while back.

> 2) Syslog-ng is configured pretty much as before, except that I have its
> systemd USE flag turned on, which builds it with journald support so the
> two cooperate a bit better.
[...]
> Meanwhile, for logs beyond one session, I still prefer traditional text-
> based logs, tho I suppose that's partly because that's what I'm most used
> to.  But there's several advantages to it that I see:

Personally, I don't mind the binary format, since the engineering advantages
convinced me (constant-time access, lots of meta-data which you can filter by,
git-inspired chain of hashes for integrity checking, etc.).

> a) If the system crashes, partially corrupted in-the-crash text logs can
> be of at least some use after a reboot.  Binary journals, not so much.

As Rich mentioned, and as I remember reading myself (on the btrfs ML?), the
journal will do its best to read back from corrupted logs and is supposedly
pretty resilient.

> b) As far as I could find, journald has absolutely no mechanism to filter
> incoming (pre-storage) log entries, while syslog-ng has had (pre-storage)
> filter mechanisms for ages.  Journald seems to have all sorts of outgoing
> post-storage filter-on-output options, some of which are pretty nifty,
> but there have been several times in the past where logs filled up with
> "noise", thousands of repeat entries for something or other that occurs
> regularly, say once a minute, that I simply do not care about and don't
> want in my logs at all, but which I can't directly stop at the source for
> whatever reason.  One of them was a kernel bug that triggered for a few
> kernel cycles, then eventually went away, another was a ksysguard
> triggered message, with that ksysguard graph updating every second, etc.
> Unfortunately, while journald makes it easy to filter these out on
> output, I found no way of preventing their appearance in the journal at
> all.
>
> But syslog-ng lets me dump them without ever actually logging them, or
> route them to a different log file if I prefer, keeping my primary logs
> clean. =:^)

That is a very good point, I hadn't thought of that.

However, I think that I prefer the post-filtering approach, since it is in
principle the more flexible one.  At work I sometimes mix the logs from
different units (e.g., units A and B, and then units B, C, and D).

But I realised today that journalctl has no negation operator!  There is,
however, an RFE for it.

But regardless of what you use, I think that the worst offenders are services
that write logs themselves (I'm looking at you, samba).

> As a counter-point, however, at least journald does compression to some
> degree, so tens of thousands of identical or nearly identical messages
> don't take up as much room as you might expect as they compress very
> well. =:^)  I guess after the first instance, dups basically consist of a
> timestamp and a pointer to the first message.  So they don't take much
> space.  Which is fortunate when some "noise" message starts appearing
> once a second... and you're storing it in tmpfs, thus in memory!
> Obviously persistent logs would be similarly space efficient, but I still
> don't like the idea of not being able to filter out the noise, BEFORE it
> hits permanent storage, so I just make sure it /doesn't/ hit permanent
> storage. =:^)

Yeah, I think that's a good use case for pre-filtering: an overly noisy service
whose logs you purposely want to ignore (for example, because you're still
testing it - but then again, for that particular case you could probably use
systemd-nspawn and log to an isolated journal).

> c) I use btrfs for my primary filesystems, and btrfs and journald's
> binary-format journals don't play so well together.  FWIW, the latest
> systemd-219 has lots of improvements for btrfs users, including some new
> features that are btrfs-specific (which I never-the-less don't expect to
> use right away as they're mostly VM/container oriented and I don't do
> much of that sort of thing, but I may well eventually use them), as well
> as some changes that should make journaling on btrfs work somewhat
> better.  So this point should be to some degree eliminated after that
> stuff stabilizes tho part of the fix is journald setting nocow on the
> journal files and that has other implications I'll skip further
> discussion of here.  However, by limiting journald to tmpfs only, I still
> get the best benefits of systemd/journald integration (like status
> including the last few log messages as mentioned above), while
> eliminating the entire class of btrfs/journald issues.  Best of both
> worlds. =:^)

Well, I'm on an SSD, but even on the laptop I haven't noticed any performance
issues (yet).  Then again, I use autodefrag, so that probably helps.

What's funny though is that the systemd news file
(http://cgit.freedesktop.org/systemd/systemd/tree/NEWS) occasionally
refers to non-btrfs file systems as "legacy file sysetms".  At least, as a
btrfs user I think it's funny :) .

> d) Tho I don't do anything with it in my syslog-ng config here, syslog-ng
> can actually make use of some of the extra information journald passes to
> it, including trusted per-app filtering and routing.  One of the problems
> that the old syslog approach had was that anything that could log
> messages could claim to be anything it wanted, and syslog really couldn't
> tell whether it was telling the truth or not, all it could do is sort and
> report the messages based on what was claimed.  A big bullet-point
> feature of systemd/journald is that because systemd enforces service
> separation via control-classes and the like, there's a much stronger
> guarantee that what systemd/journald says is service X, is REALLY service
> X and not service Y!  But that's not limited to journald.  Syslog-ng has
> for several versions now had the ability to filter, route and report on
> out-of-log-channel information such as the extra properties that journald
> associates with each message, that make features such as systemctl status
> reporting the last few log messages from that service possible, because
> systemd/journald actually tracks it, AND passes it on to other loggers as
> well now, providing they're built to use the API provided by the systemd
> library I mentioned above.  And again, for gentooers, syslog-ng is built
> with that API, against that library, based on USE=systemd.  So once
> that's there, if desired, you can do all sorts of fancy filtering/log-
> routing/etc based on information such as the application/service name,
> that systemd passes along to syslog-ng.
>
> And again, with syslog-ng, this can be done BEFORE the actual logging if
> desired, instead of logging everything and filtering only on output, as
> journald does. =:^)

That's actually pretty cool :) . You're basically using the journal as a log
accumulator and syslog-ng for routing and post-processing its messages.

Whiles that's not how I want to do things, I still appreciate how cool that
is :) (and the fact that the journal can be used that way).

> e) I'm simply more comfortable dealing with text-based logs, since I'm
> used to them and can use whatever tools I want on them, as well as being
> able to split/sort/route to multiple log files in as simple or complex a
> setup as I want. =:^)
>
> Summary:
>
> For me, splitting up logging duties so journald does the collection, but
> only does session journaling to tmpfs, while syslog-ng still handles the
> "persistent" logging, gives me the best of both worlds.  I love systemd/
> journald features such as log-messages-in-status and having seen how
> useful it is, would seriously find it hard to do without.  But at the
> same time, I can't bring myself to accept persistent logging without the
> ability to filter out the noise before it hits persistent storage, and
> journal storage on btrfs is problematic anyway, with both problems nicely
> solved by handing off to syslog-ng for my persistent storage needs while
> keeping journald for same-session use, so that's exactly what I do. =:^)

Right :) .

Greetings
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup