Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 2/3/19 5:37 AM, Rich Freeman wrote:
> Nothing wrong with that approach.  I use systemd-nspawn to run a bunch
> of containers, hosted in Gentoo, and many of which run Gentoo.  However,
> these all run systemd and I don't believe you can run nspawn without a
> systemd host (the guest/container can be anything).  These are containers
> running full distros with systemd in my case, not just single-process
> containers, in my case.  However, nspawn does support single-process
> containers, and that includes with veth, but nspawn WON'T initialize
> networking in those containers (ie DHCP/etc), leaving this up to the guest
> (it does provide a config file for systemd-networkd inside the guest if
> it is in use to autoconfigure DHCP).

ACK

That makes me think that systemd-nspawn is less of a fit for what I'm 
wanting to do.

> I'm not exactly certain what you're trying to accomplish, but namespaces
> are just a kernel system call when it comes down to it (two of them I
> think offhand).  Two util-linux programs provide direct access to them
> for shell scripts: unshare and nsenter.  If you're just trying to run a
> process in a separate namespace so that it can use veth/etc then you could
> probably initialize that in a script run from unshare.  If you don't need
> more isolation you could run it right from the host filesystem without
> a separate mount or process namespace.  Or you could create a new mount
> namespace but only modify specific parts of it like /var/lib or whatever.

That's quite close to what I'm doing.  I'm actually using unshare to 
create a mount / network / UTS namespace (set) and then running some 
commands in them.

The namespaces are functioning as routers.  I have an OvS switch 
connected to the main / default (unnamed) namespace and nine (internal) 
OvS ports, each one in a different namespace.  Thus forming a backbone 
between the ten network namespaces.

Each of the nine network namespaces then has a veth pair that connects 
back to the main network namespace as an L2 interface that VirtualBox 
(et al) can glom onto as necessary.

This way I can easily have nine completely different networks that VMs 
can use.  My main home network has a route to these networks via my 
workstation.  (I'm actually using routing protocols to distribute this.)

So the main use of the network namespaces is as a basic IP router. 
There doesn't /need/ to be any processes running in them.  I do run BIRD 
in the network namespaces for simplicity reasons.  But that's more 
ancillary.

I don't strictly need the mount namespaces for what I'm currently doing. 
  That's left over from when I was running Quagga and /needed/ to alter 
some mounts to run multiple instances of Quagga on the same machine.

I do like the UTS namespace so that each ""router has a different host 
name when I enter it.

Maybe this helps explain /what/ I'm doing.  As for /why/ I'm doing it, 
well because reasons.  Maybe not even good reasons.  But I'm still doing 
it.  ¯\_(ツ)_/¯  I'm happy to discuss this in a private thread if anyone 
is really curious.

> People generally equate containers with docker but as you seem to get
> you can do a lot with namespaces without basically running completely
> independent distros.

Yep.  I feel like independent distros, plus heavier weight management 
daemons on top are a LOT more than I want.

As stated, I don't really /need/ to run processes in the containers.  I 
do because it's easy.  The only thing I /need/ is the separate IP stack 
/ configuration.

> Now, I will point out that there are good reasons for keeping things
> separate - they may or may not apply to your application.  If you just
> want to run a single daemon on 14 different IPs and have each of those
> daemons see the same filesystem minus /var/lib and /etc that is something
> you could certainly do with namespaces and the only resource cost would
> be the storage of the extra /var/lib and /etc directories (they could
> even use the same shared libraries in RAM, and indeed the same process
> image itself I think).

Yep.

> The only gotcha is that I'm not sure how much of it is already done, so
> you may have to roll your own.  If you find generic solutions for running
> services in partially-isolated namespaces with network initialization
> taken care of for you I'd be very interested in hearing about it.

I think there are a LOT of solutions for creating and managing 
containers.  (I'm using the term "container" loosely here.)  The thing 
is that many of them are each their own heavy weight entity.  I have yet 
to find any that integrate well with OS init scripts.

I feel like what I want to do can /almost/ be done with netifrc.  Or 
that netifrc could be extended to do what (I think is) /little/ 
additional work to do it.

I don't know that network namespaces are strictly required.  I've been 
using them for years.  That being said, the current incarnation of 
Virtual Routing and Forwarding (VRF) provided by l3mdev seems to be very 
promising.  I expect that I could make VRF (l3mdev) do what I wanted to 
do too.  At least the part that I /need/.  I'm not sure how to launch 
processes associated with the VRF (l3mdev).  I'm confident it's 
possible, but I've not done it.

But, even VRF (l3mdev) is not supported by netifrc.  I feel like the 
Policy Based Routing (PBR) is even a kludge and largely consists of 
(parts of) the ip / tc commands being put into the /etc/conf.d/net file.

I feel like bridging / bonding / VLANs have better support than PBR 
does.  All of which are way better supported than VRF (l3mdev) which is 
better supported than network namespaces.

Though, I'm not really surprised.  All of the init scripts that I've 
seen seem to be designed around the premise of a singular system and 
have no knowledge that there might be other (virtual) systems.  What 
little I know about Docker is that even it's configuration is singular 
system in nature and still only applies to the instance that it's 
working on.  I've not seen any OS init scripts that are aware of the 
fact that they might be working on other systems.  I think the closest 
I've seen is FreeBSD jails.  But even that is separate init scripts, 
which are again somewhat focused on the jail.

I need to do some thinking about /what/ /specifically/ I want to do 
before I start thinking about /how/ to go about doing it.

That being said, I think it would be really nice to have various 
interfaces tagged with what NetNS they belong to and use the same 
net.$interface type init scripts for them.