Re: [gentoo-user] Re: Is gnome becoming obligatory?

[Marc Joliet]

Am Donnerstag, 14. Dezember 2017, 16:52:59 CET schrieb Ian Zimmerman:
> On 2017-12-14 11:57, Marc Joliet wrote:
> > I could list specific features of systemd that I like and make use of
> > (such as socket activation, autofs integration, user units, nspawn, or
> > the journal), but thinking about it, it's a "more than the sum of its
> > parts" kind of deal.  Managing a system with systemd is just overall
> > pleasant for me.
>
> I am probably not the only one who would still dearly like such a
> detailed list, from someone I don't see as biased to start with.  I
> understand this is a drain on your time, so I'll understand if you
> decline.

I don't mind per se, but whether I'm inclined to do so depends on the attitude of who's asking.  If I'm asked politely the way you did then I don't mind at all, in fact it can be quite pleasant :) (and it also serves to refresh my memory).

Before I begin, to avoid making this sound all rainbows and glitter, I will mention one thing that annoys me about systemd:  regressions.  None of them have been horrible, and I think I was only ever hit by 2 or 3 over the course of by now several years, but they're still annoying.  The one I'm currently waiting on is the systemd-run bug when both cgroup V1 and V2 trees are mounted.  I could work around it (by deactivating cgroup V2 support or some such), but I don't miss systemd-run so much that I could be bothered to.  Regardless, I wish upstream would handle them better.

Also, this is just my perspective as a hobbyist (although I've used systemd in a professional context).  I could recommend some presentations that provide a different perspective (e.g., by a company that uses systemd in IP cameras, where the cgroups based resource management features of systemd came in very handy).  There's also a presentation by Klaus Knopper of Knoppix fame that is overall more negative, but still mostly fair -- though IMHO not completely -- and as I recall one of the better criticisms I've seen.  His perspective is that of somebody trying to provide a Linux OS for computers destined for poorer countries, where he has to deal with less capable hardware (i.e., no SSDs, booting from CD/DVD).

(Also also, sorry in advance for the wall of text, this kind of, uh, spun out of control.  Sorry also to Ian and Peter for continuing to feed the monster thread ;-) .)

Alright, so here's what I can think of now, starting off with what I listed above and then continuing with anything else I can remember.  This includes more or less verbose descriptions of what the features bring to the table, including specifically how they help *me*.

1.) Socket activation sounds like a detail, but it has several positive side effects.  It can make dependency specifications unnecessary, thus making unit files simpler, and it increases the number of services that can start in parallel.  It also enables on-demand services á la (x)inetd, only generalised for all system services (I use SSH this way by only enabling the socket unit).  You can't use systemd without using this feature, something's bound to use it.

This is the main reason systemd can boot so fast (on flash storage, at least).  That may not be important to some people, but it is to me (and embedded projects).

2.) Autofs integration via automount units allows dynamic mounting (and optionally unmounting) of file systems, which I use on my desktop to asynchronously mount my data dump (a 2x1TB btrfs RAID1).  To be honest, I mostly did this to speed up boot time (I think I mention timings in the Email thread I referenced), but it also makes boot-up finish independent of mount failures.  On my home server (an old Mac Mini) I use it for a USB drive for the same reasons (although boot time isn't so important there).

[ Just to be clear: autofs is a Linux kernel feature, systemd just exposes it in an easy to use way.  That is, BTW, a theme with systemd. ]

3.) Personally I find user units (systemd units that run as your user) super practical.  There is a system location for them, but you can place your own in your home directory under ~/.config/systemd/user/ and have them start when you log in.  If you configure your user session -- which is basically a systemd instance running as your user -- as "lingering", you can have persistent user services that run as long as your computer is on (strictly speaking, for as long as your user session is active).  For example, my desktop looks like this:

> % systemctl --user list-units -t service -t timer -a
>   UNIT                      LOAD      ACTIVE   SUB     DESCRIPTION
>   ctags.service             loaded    inactive dead    Regenerate ctags files
>   gpg-agent.service         loaded    active   running Start gpg-agent (with SSH support)
>   mpd.service               loaded    active   running Music Player Daemon
>   newsboat.service          loaded    inactive dead    Run newsboat -x reload
> ● syncthing-inotify.service not-found inactive dead    syncthing-inotify.service
>   syncthing.service         loaded    active   running Syncthing - Open Source Continuous File Synchronization
>   ctags.timer               loaded    active   waiting Regenerate ctags files (timer)
>   newsboat.timer            loaded    active   waiting Run newsboat -x reload (timer)
>
> LOAD   = Reflects whether the unit definition was properly loaded.
> ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
> SUB    = The low-level unit activation state, values depend on unit type.
>
> 8 loaded units listed.
> To show all installed unit files use 'systemctl list-unit-files'.

So I run MPD, gpg-agent, and syncthing continuously, while ctags and newsboat are executed on a schedule via timers (a cron-like feature, see point 6 below), hence their low-level activation state "dead".

Oh, and all units' output gets collected in the journal (see point 5 below), which doesn't work with the alternatives (e.g., ~/.xprofile, which I used to use).

4.) I use nspawn occasionally for running a Gentoo amd64 container for testing asciidoc (I wound up proxy maintaining it).  It's kind of a chroot on steroids, hence you can use it in place of chroot, but you can also do things like run containers as services (i.e., there's such a thing as an nspawn unit), but I don't use that feature.  Just as an example, the container I use is configured like this:

% cat /etc/systemd/nspawn/gentoo-amd64-systemd.nspawn 
[Files]
BindReadOnly=/home/marcec/projects/gentoo/:/home/marcec/gentoo/
BindReadOnly=/usr/portage/distfiles/:/usr/portage/ro_distfiles/
TemporaryFileSystem=/var/tmp/portage/

(I bet you never heard of ro_distfiles!  I learned about it while setting up the container (look up PORTAGE_RO_DISTDIRS in make.conf(5)).)

So this is pretty much like docker, which makes sense, since they use the same underlying Linux kernel features.  Systemd simply enables the treatment of containers as services (including the ability to connect to a systemd instance running inside a container, in the event that the container uses systemd as its service manager).

5.) The journal is another element of systemd that one could write a lot about.  My usage is mostly limited to inspecting the state of a service ("systemctl status <unit>" includes the last 10 lines of log output by default) and looking for stuff in a specific time range.  However, it has a few advantages over classic syslog implementations:

- By default it captures all stdout and stderr of service units, so you can't miss anything.
- It groups all log output of a service, regardless of how many processes it consists of, i.e., you can't miss anything (I'm thinking of naive greps here).
- While you can just use grep (and I sometimes do), it's often better to use the builtin search functionality, e.g., you can output journal entries corresponding to a specific kernel device (not useful to me, personally, but illustrates that there are a lot of filters you can use).
- As the prior point illustrates, journal entries consist of a *lot* of metadata, whereas with syslog, you have a quasi-structured text file that contains less information.
- It helps unclutter /var/log/ ;-) .
- Probably more that I can't think of right now.

If you don't care for it, you can also turn off persistent journal storage and run a syslog daemon instead, or in parallel (you can't completely turn off the journal, though, as it is still required for capturing stdout and stderr).

6.) I also happen to like timers a lot, because I can forego a cron daemon, and I happen to like the "systemctl list-timers" output very much.  But really, you can just keep using cron, it's not a world shattering feature for me.  It does have the advantage of allowing dependencies to other units, though, since timer units are units like anything else in systemd.  This can be really powerful, since, e.g., devices and mount points have corresponding unit types.

Actually, the main advantage of using timer units for me is that I can put them in git and synchronise them across different computers, which I couldn't really do with cron, because crontabs live somewhere in /var/ (right? Or was that only the root crontab?).  Also, since timers trigger service units, their output goes to the journal, too.

7.) Use of ACLs.  This is just a detail, but I like that my user looks like this:

% id
uid=1000(marcec) gid=100(users) Gruppen=100(users),10(wheel),35(games),1019(realtime)

So I'm left with wheel (used by sudo), games (which will go away with time, what with the games eclass being deprecated), and realtime, which doesn't deal with device access anyway.

The point is, I don't need to add myself to the right groups to gain access to certain devices, because systemd adds the right ACLs to the right device nodes once you log in (this is one of the jobs of logind, BTW), e.g.:

% getfacl  /dev/snd/seq 
getfacl: Removing leading '/' from absolute path names
# file: dev/snd/seq
# owner: root
# group: audio
user::rw-
user:marcec:rw-
group::rw-
mask::rw-
other::---

This is specifically to support multi-user systems, where the current active user gets access to required devices, but I like not having to bother with adding myself to the right groups most of the time.  Of course, there are situations in which it still makes sense to add yourself to a specific device group.  For example, on my home server my user is still part of the "audio" group, because otherwise I would never be granted the proper ACLs since there's no persistent login session.  That would be a problem for pulseaudio, and hence for MPD.

ACLs are also used for users' persistent journal storage, i.e., you can read your user's journal entries via "journalctl --user" and "--user-unit", but not the system journal.

8.) Nothing super important, but I like how you can see an overview of the differences between the default configuration files and your own overrides with systemd-delta:

> % systemd-delta
> [OVERRIDDEN] /etc/tmpfiles.d/tmp.conf → /usr/lib/tmpfiles.d/tmp.conf
>
> --- /usr/lib/tmpfiles.d/tmp.conf        2017-12-05 17:11:28.999238832 +0100
> +++ /etc/tmpfiles.d/tmp.conf    2016-03-10 23:39:49.343619664 +0100
> @@ -8,15 +8,12 @@
>
>  # See tmpfiles.d(5) for details
>
>  # Clear tmp directories separately, to make them easier to override
>
> -q /tmp 1777 root root
> -q /var/tmp 1777 root root
> +# undo the Gentoo patch 226-noclean-tmp.patch
> +v /tmp 1777 root root 10d
> +v /var/tmp 1777 root root 30d
>
>  # Exclude namespace mountpoints created with PrivateTmp=yes
>  x /tmp/systemd-private-%b-*
>  X /tmp/systemd-private-%b-*/tmp
>  x /var/tmp/systemd-private-%b-*
>  X /var/tmp/systemd-private-%b-*/tmp
>
> -
> -# Remove top-level private temporary directories on each boot
> -R! /tmp/systemd-private-*
> -R! /var/tmp/systemd-private-*
>
> [EXTENDED]   /usr/lib/systemd/system/irqbalance.service → /etc/systemd/system/irqbalance.service.d/override.conf
> [EXTENDED]   /etc/systemd/system/portage-sync.service → /etc/systemd/system/portage-sync.service.d/network.conf
> [EXTENDED]   /usr/lib/systemd/system/chronyd.service → /etc/systemd/system/chronyd.service.d/arguments.conf
> [EXTENDED]   /usr/lib/systemd/system/btrbk.service → /etc/systemd/system/btrbk.service.d/deps.conf
> [EXTENDED]   /usr/lib/systemd/system/cpupower-frequency-set.service → /etc/systemd/system/cpupower-frequency-set.service.d/00gentoo.conf
> [EXTENDED]   /usr/lib/systemd/system/btrbk.timer → /etc/systemd/system/btrbk.timer.d/schedule.conf
>
> 7 overridden configuration files found.

Aha, so this tells me that I need to update my tmp.conf override.

Oh, also you can override parts of system unit files by creating appropriate *.conf files in <unit>.d/ directories.

9.) systemd-tmpfiles provides a unified way of creating users, groups, directories, etc. required by the system in some way (it also has a cleanup feature like app-admin/tmpwatch).  This makes more sense when you realise that these files are (also) supposed to be provided by upstream projects, just like unit files.  So you end up with one canonical definition of what directories, users, etc. a project requires at run-time.  This is a feature that makes software developer's lives easier.  It's also useful for containers and stateless systems, though.

(OpenRC gained support for this via sys-apps/opentmpfiles.)

10.) One last thing: btrfs integration.  If the underlying file system is btrfs, systemd will make use of subvolumes (the "v" and "V" specifiers in tmpfiles files) and qgroups ("q" and "Q"), and will exploit its copy-on-write semantics, e.g., when working with containers (which docker does, too, I believe).  The former two are nice to have and can help with snapshotting, the latter simply saves space.

11.) No, sorry, *one* more thing: generators.  These are programs that generate systemd units at run-time, which is, e.g., used by sys-apps/gentoo-systemd-integration for generating service units from the files in /etc/local.d/, or by systemd for generating mount units from fstab, or (my favourite) by sys-process/systemd-cron for generating timer units from files in /etc/cron.*/.

OK, I'm done now.

I hope that the "integrated toolkit" nature of systemd came across here, since that's pretty much what systemd is: a toolkit for managing Linux systems, to which belongs a service manager (init system), system logger, periodic job execution, and various other bits and piece that often enough supplement each other to yield a system that is, in my view, more than the sum of its parts.

> This is also equally directed at Neil, who also posted a similar
> abbreviated list of features.

I'm curious about what he, and others in general, would say, too.  I can imagine some people who might have a more differentiated opinion.

HTH
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup