Re: [gentoo-user] technical review of systemd

> On Sat, Feb 22, 2014 at 6:16 PM,  <thegeezer@×××××××××.net> wrote:
>> OK so because of how much time has been spent arguing about systemd with
>> little technical content, i've spent some time on the freedesktop site
>> reading Lennart's blog and also going through the source to find answers
>> to my questions about the socket activator.
>> i've also been going through the man pages of netctl too and am
>> horrified
>> at the lack of what i would call enterprise features.
>
> networkd (netctl is just the command-line front-end) is not intended
> for enterprise; it's for little servers where you only need static IPs
> or simple bridges. For desktops/laptops, you are supposed to keep
> using NetworkManager/connman/whatever you used before. For complex
> network setups, you need *a* network manager (not necessarily
> NetworkManager).
>
>> this is by no means a definitive list.
>> I just thought that i would share what i had found.
>> please correct me if i am wrong in any of these.
>> please add to the list for technical items only.
>
> I find it a very impartial and objective review; thank you very much!
>
>> thanks!
>>
>> pros
>> 1.very modular, everything can be disabled though not removed
>> 2.socket based activator allows restart of services with no service
>> interruption
>> 3.if activator.c is used for this, then the code is actually pretty
>> clean
>> using supplied sd-daemon.c simplifies sockets for daemons and also adds
>> extra watchdog features
>> 4.can disable socket based activation according to Canek, but i can't
>> find
>> how.
>
> You use a .service unit file instead of a .socket unit file. That's it.
>

thanks good to know that is all you need

> For OpenSSH, for example, you can enable sshd.service[1], and then the
> SSH daemon works as it does in OpenRC. If you instead enable
> sshd.socket[2], then the daemon will start on demand.
>
> You don't have to *disable* anything; you choose how do you want to
> use your services (if the services provide both ways, like OpenSSH
> does).
>
>> 5.fschecking mounts and logging output (though how for corrupt /
>> notsure)
>
> Corrupt filesystems or logs?
>

logs.  currently if fsck runs anywhere on boot i get zero log about what
was done, so i prefer to do this on a running system.  / is obviously
special, so this is a pro that fsck is logged, but of course if / has
issue i'm not sure what systemd would do other than drop you to emergency

>> 6.auto-gettys allows for lower numbered X windows by default for e.g.
>> multiseat and dynamic serial ttys
>> 7.clever logging, including from nspawned containers' logs and
>> distributed
>> for enterprise
>> 8.nspawning using filename namespaces
>> 9.systemctl kill <service> -- killing service and all forks and spawn
>> cgtop -- top with cgroups
>> 10.much easier to define resource limitations per service
>>
>> cons
>> 1.new tools to learn, new gotchas to learn.
>> 2.yet to go through systemd source to find out how modular or not it is.
>
> While it tries to be modular where it can, systemd prefers simple code
> and integrated solutions. Modularity is not going to be one of its
> strong points.
>
>> 3.not clear how the socket activator works, the code activator.c appears
>> to be to _test_ activation only, with activator code being elsewhere.
>> if
>> it is used then you would have one process running for each port it is
>> virtually listened to.
>
> It's been a while since I've read the source code, but it isn't in
> src/activate/activate.c[3]?

ok so it does look like it would have a systemd-activate process for each
socket being activated on behalf of a service. that makes me feel better
than one process doing all of them. perhaps someone using service
activation can do a 'ps aux' to confirm?

>> 4./etc/machine-id   because hostname and node id in the <cluster of your
>> choice> are not enough.
>
> The idea is that machine-id is as unique as reasonable to ask. I'm not
> overly happy with it, too, but that's the justification.
>
> Imagine thousands of virtual machines running services, and you want
> to coalesce all their journal logs in a central server. With
> machine-id, you don't need to worry even to change the default
> "localhost" for your throwaway VMs, you can detect the different logs
> immediately (machine-id should be generated at OS install time; for
> rolling distros, I think they generate it if when installing systemd
> is not available.)
>
>> 5./fsck.options gives more options than "auto""force""skip" on reboot
>> 6.requiring logging tools in rescue cds in order to view logs
>
> Yeah, that's a drag. However, you *can* run rsyslog (or syslog-ng)
> alongside the journal, and have the best of both worlds. Or you can
> automatically send the journal logs to a central server designed for
> that purpose only.
>
>> 7.chroots no longer work. forcing use of nspawn to ensure environment
>> set
>> up correctly.
>
> I'm sorry, chroot doesn't work? First time I heard about it. While
> systemd-nspawn is a gazillion times better than a simple chroot, you
> *can* still use a chroot if you so desire. Where did you found that
> chroot doesn't works?

agreed nspawn is better due to the filesystem namespaces.  chroot itself
works, but the environment doesn't so it is effectively broken.  full
explaination from lennart's blog [5]  This is quite old so i don't know if
this has been fixed, seems unlikely based on what he described

>> 8.strange gotchas: that because of socket/dbus etc activation you have
>> to
>> disable a service first, then stop it in case it is then restarted in
>> the
>> background
>
> You only need to stop the socket too. You can do it at the same time:
>
> systemctl stop avahi-daemon.service  avahi-daemon.socket
>

good to know

>> 9.the new deal breaker for me is the networking.
>> for anything remotely complex (i.e. two IP addresses on an interface
>> woo),
>> need to use netctl.
>>  a.which doesn't support vlan naming types i.e. padding zeroes
>>  b.doesn't appear to support gre keys
>>  c.doesn't appear to support multiple routing tables
>>  d.doesn't appear to support "ip rule"
>>  e.doesn't have lacp support for bonding
>>  f.there is the option for running a script in PRE and POST UP but...no
>
> networkd (again, netctl is the command-line front-end) is not for
> enterprise networks; on the contrary, is for the trivial cases. For
> example, in a little web server I administer I have:
>
> $ cat /etc/systemd/system/network.service
> [Unit]
> Description=Static network service
> After=local-fs.target
> Before=network.target
> Documentation=man:ifconfig(8)
> Documentation=man:route(8)
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStart=/bin/ifconfig enp2s12 192.168.1.2 broadcast 192.168.1.255
> netmask 255.255.255.0 up
> ExecStart=/bin/route add default gw 192.168.1.1 enp2s12
>
> [Install]
> WantedBy=multi-user.target
>
> (Yeah, I know, I should switch to ip, I'm sorry, I haven't had the time
> yet).
>
> I'm going to get rid of this trivial network.service unit file when
> 209 (or better 210) hits Gentoo. Cases like this are the use-cases for
> networkd.
>

what i don't understand is if you look at how openRC does it, it only
really cares about up/down events and the /etc/conf.d/net is very
comprehensive, in part because it passes everything to iproute2 to handle,
the only thing i can't do without an additional shell script is tc qdiscs.
i don't need or want a network manager, just need something that applies
confs on startup / stop of interfaces.  I'm a little surprised that there
isn't an iproute2 .service file

reading through your example, in fact this is preferrable to me than using
a network manager but using iproute2.  I would rather you keep this
example in, and have this shown on the wiki or somewhere as this neatly
resolves my network concern.

>> 10.strange gotchas: /tmp being tmpfs using up to 50% ram.   unless
>> mounted
>> in fstab
>
> That doesn't have nothing to do with systemd: from man:mount(8):
>
> """
> Mount options for tmpfs
>        size=nbytes
>               Override  default  maximum  size  of  the  filesystem.
> The size is given in bytes, and rounded up to entire pages.  The
> default is half of the memory. The size parameter also
>               accepts a suffix % to limit this tmpfs instance to that
> percentage of your physical RAM: the default, when neither size nor
> nr_blocks is specified, is size=50%
> """
>
> systemd just mounts the tmpfs; the default *is* 50%.
>

I was just highlighting because someone could be putting systemd on
embedded and wondering why memory usage increased over time. no issue if
you have 8GB+, but if you have 256MB is an issue, and if you have less you
are counting every byte.

>> 11.strange gotchas: logging is volatile by default _unless_
>> /var/log/journal exists, when it becomes persistent due to the "auto"
>> default.
>
> It's by design; a distro can decide if it creates /var/log/journal or
> nor for its users at install time. RHEL probably will; Ubuntu
> (desktop) probably wont.
>
>> 12.transitions into systemd are non-trivial.
>
> No, they are not.

i'm not talkign about new installs.  for me the transition is not easy. 
most of my gentoo boxes are headless, and many are in far flung places.  a
migration into systemd would be a nightmare of repeatedly checking items
to ensure the reboot would succeed remotely, and have a working network
connection and working sshd.  in a previous post i mentioned that i would
need to do the following, requiring at least two reboots from gentoo wiki
[4]
.. new kernel options
.. reboot
.. new grub2 command line
.. install dbus (use=-systemd) _then_ systemd
.. create network.service as you have shown
.. add grub secondary entry with systemd init and reboot
.. would be nice to use an import for localed and hostnamed and timedated
.. importing openrc services and runlevels to targets
.. adjust syslog and systemd to work together
also there are 'workarounds' for lvm early masking -- i use LVM everywhere
and i wouldn't know which location would need this or if my boot was
broken for another reason.

a trivial migration would be
# emerge systemd (auto removes openrc)
# systemd-import-openrc (including network issues as mentioned in this post)
# reboot

>
>> my own conclusions
>> systemd seems to be excellent for a desktop
>> good for _new_ instances of service VMs. I say new because of the large
>> job of transitioning away from openRC, but all the watchdog and better
>> resource management will help to pack datacentres.
>
> I agree.
>
>> It would also be good for big iron running many services because of
>> this,
>> but then i thought everyone was using small fast service specific gentoo
>> VMs to compartmentalise anyway --- or was that just me?
>
> I think Gentoo users come in all sizes and shapes ;)
>
>> Unless I have completely got netctl wrong it is terrible for a
>> firewall/router scenario, or being the host server for LXC containers
>> which is a shame because resource management built in to service control
>> combined with say docker.io would be a great combination; as long as you
>> don't use custom VLAN settings.
>
> I think you got it wrong; it's just to fill the needs of simple
> networks when you don't want all the bells and whistles of a big
> network manager (not necessarily NetworkManager).
>
> BTW, Kay and Lennart Ok'd the inclusion of networkd (and they had
> thought it was missing from systemd), but it was written by Tom
> Gundersen, primarily.
>

I think actually that i would just create an iproute2 service file for
each device.  so have no network manager at all and have the service file
start after the interface.   iproute2.enp2s0.service etc
in there i would have "exec=/sbin/ip rule add" etc
i'd also have "exec=/sbin/ip addr add" etc

>> As Gentoo is a meta-distro (says Larry the Cow
>> http://www.gentoo.org/main/en/about.xml) and a rolling release distro,
>> I'm
>> all for choice, but I would sincerely hope that unlike all of the other
>> distributions from Arch to Ubuntu systemd is not adopted by default as
>> udev and baselayout transitions were bad enough.
>
> As a systemd supporter, at some point in the long, long future, I
> would be more than happy if systemd was added to the handbook as
> "secondary supported init system" in its own section. I'm completely
> fine with OpenRC as the default.
>

that would be the ideal way forwards

> Also (and I plan to work on this in the future), I would like to have
> LiveCDs and stages with systemd installed (not necessarily hosted in
> the Gentoo infrastructure), because is works really nicely install
> Gentoo from systemd-nspawn instead of a chroot.
>

actually I agree - even if it is only to get journactl on a bootable cd!

> Someone would have to do that, though; I hope to help with that in the
> future.
>

to end on a merry note [6] and kind of why i wrote this

>> I will however be installing a systemd desktop in a vm to play properly.
>> YMMV
>
> Thanks again for a succinct, impartial and objective analysis of systemd.
>

very welcome, hope this helps others

> Regards.
>
> [1]
> http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-misc/openssh/files/sshd.service?view=markup
> [2]
> http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-misc/openssh/files/sshd.socket?view=markup
> [3] http://cgit.freedesktop.org/systemd/systemd/tree/src/activate
> --
> Canek Peláez Valdés
> Posgrado en Ciencia e Ingeniería de la Computación
> Universidad Nacional Autónoma de México
>
>

[4] http://wiki.gentoo.org/wiki/Systemd
[5] http://0pointer.de/blog/projects/changing-roots.html
[6]
http://allpoetry.com/poem/9326367-Everybody--Anybody--Somebody--Nobody-and-Someone-Else-by-YoungDumbCrazy