Re: [gentoo-user] Can I use containers?

[Grant Taylor]

On 5/18/19 5:49 PM, Rich Freeman wrote:
> I'd be interested if there are other scripts people have put out
> there, but I agree that most of the container solutions on Linux
> are overly-complex.

Here's what I use for some networking, which probably qualifies as 
extremely light weight ""containers.

Prerequisite:  Create a place for the name spaces to anchor:

    # Create the directories to contain the *NS mount points.
    sudo mkdir -p /run/{mount,net,uts}ns

You can use any path that you want.  —  I do a lot with iproute2's 
network namespaces (which is where this evolved from), which use 
/run/netns/$NetNSname.  So I used that as a pattern for the other types 
of namespaces.  Adjust as you want.  —  What I'm doing is interoperable 
with iproute2's netns command.

Per ""Container:  Create the ""Containers mount points:

    # Create the *NS mount points
    sudo touch /run/{mount,net,uts}ns/$ContainerName

Start the actual namespaces:

    # Spawn the lab# NetNSs.
    unshare --mount=/run/mountns/$ContainerName 
--net=/run/netns/$ContainerName --uts=/run/utsns/$ContainerName /bin/true

Note:  The namespaces don't die when true exits because they are 
associated with a mount point.

Tweak the namespaces:

    # Set the lab# NetNS's hostname.
    nsenter --mount=/run/mountns/$ContainerName 
--net=/run/netns/$ContainerName --uts=/run/utsns/$ContainerName 
/bin/hostname $ContainerName

I reuse this command calling different binaries any time I want to do 
something in the ""container.  Calling /bin/bash (et al.) enters the 
container.

I've created a wrapper script (nsenter.wrapper) that passes the proper 
parameters to nsenter.  I've then sym-linked the container name to the 
nsenter.wrapper script.  This means that I can run "$ContainerName 
$Command"  or simply enter the container with $ContainerName.  (The 
script checks the number of parameters and assumes /bin/bash if no 
command is specified.

I think it's ultimately extremely trivial to have a ""container 
(glorified collection of name spaces) to do things I want with virtually 
zero disk space.  Ok, ok, maybe 1 or 2 kB for the script & links.

Note:  Since I'm using the mount name space, I can have a completely 
different mount tree inside the ""container than I have outside the 
container / on the host.  I'm not currently doing that, but it's 
possible to change things as desired.

> I personally use nspawn, which is actually pretty minimal, but it
> depends on systemd, which I'm sure many would argue is overly complex.
> :)  However, if you are running systemd you can basically do a
> one-liner that requires zero setup to turn a chroot into a container.

As much as I might not like systemd, if you have it, and it reliably 
does what you want, then I see no reason to /not/ use it.  Just 
acknowledge it as a dependency on your solution, which you have done. 
So I think we're cool.

> On to the original questions about mounts:
>
> In general you can mount stuff in containers without issue.  There are
> two ways to go about it.  One is to mount something on the host and
> bind-mount it into the container, typically at launch time.  The other
> is to give the container the necessary capabilities so that it can
> do its own mounting (typically containers are not given the necessary
> capabilities, so mounting will fail even as root inside the container).

Given that one of the uses of containers is security isolation (such as 
it is), I feel like giving the container the ability to mount things is 
less than a stellar idea.  But to each his / her own.

> I believe the reason the wiki says to be careful with mounts has more
> to do with UID/GID mapping.  As you are using nfs this is already an
> issue you're probably dealing with.  You're probably aware that running
> nfs with multiple hosts with unsynchronized passwd/group files can
> be tricky, because linux (and unix in general) works with UIDs/GIDs,
> and not really directly with names,

That's true for NFS v1-3.  But NFS v4 changes that.  NFS v4 actually 
uses user names & group names and has a daemon that runs on the client & 
server to translate things as necessary.

> so if you're doing something with one UID on one host and with a
> different UID on another host you might get unexpected permissions
> behavior.

Yep.  You need to do /something/ to account for this.  Be it manually 
manage UID & GID across things, or use something like NFSv4's 
synchronization mechanism.

> In a nutshell the same thing can happen with containers, or for
> that matter with chroots.

I mostly agree.  However, user namespaces can nullify this.

I've not dabbled with user namespaces yet, but my understanding is that 
they can have completely different UIDs & GIDs inside the user namespace 
than outside of it.  It's my understanding that UID 0 / GID 0 inside a 
user namespace can be mapped to UID 12345 / GID 23456 outside of the 
user namespace.  Refer to nsenter / unshare man pages for more details.

> If you have identical passwd/group files it should be a non-issue.

Point of order:  The files don't need to be identical.  The UIDs & GIDs 
need to be managed if you aren't using something like user namespaces. 
So it's perfectly valid to have a text file that is used to coordinate 
UIDs & GIDs somewhere and then use those in passw/shadow group/gshadow 
files.

> However, if you want to do mapping with unprivileged containers
> you have to be careful with mounts as they might not get translated
> properly.  Using completely different UIDs in a container is their
> suggested solution, which is fine as long as the actual container
> filesystem isn't shared with anything else.

I conceptually agree.  However I think mount namespaces combined with 
user namespaces muddy the water.  Again, refer to the nsenter / unshare 
man pages and what they refer to.

nsenter has an option for sharing something between mount namespaces.  I 
have no idea what it does, much less how it does it.  I suspect that the 
kernel mounts it once (maybe not visible from anywhere else) and then 
bind-mounts it to multiple locations for visibility / access.

> That tends to be the case anyway when you're using container
> implementations that do a lot of fancy image management.  If you're
> doing something very minimal and just using a path/chroot on the host
> as your container then you need to be mindful of your UIDs/GIDs if
> you go accessing anything from the host directly.

UID & GID management is important.  /Something/ should be doing it.

> The other thing I'd be careful with is mounting physical devices in
> more than one place.  Since you're actually sharing a kernel I suspect
> linux will "do the right thing" if you mount an ext4 on /dev/sda2 on
> two different containers, but I've never tried it (and again doing
> that requires giving containers access to even see sda2 because they
> probably won't see physical devices by default).

Seeing as how the containers are running under the same kernel, there is 
no actual need for the file system to be mounted multiple times. 
Instead the kernel would mount it and present it, much like a bind 
mount, to multiple containers for access.

Think along the lines of opening and working with a file system as a 
separate process from where it's presented for access.  Conceptually not 
that dissimilar to a hard link that has multiple representations of a 
file in multiple locations on the same file system.  (It's not a perfect 
analogy, but I hope that makes sense.)

> In a VM environment you definitely can't do this, because the VMs
> are completely isolated at the kernel level and having two different
> kernels having dirty buffers on the same physical device is going
> to kill any filesystem that isn't designed to be clustered.

Technically, you can usually get away with doing this.  But the mounts 
need to be read-only.  But I STRONGLY suggest that you NOT do this to a 
non-cluster aware file system.

I have colleagues that supported systems RO mounting an Ext file system 
this way.  It worked okay when it was used as a RO library.  The problem 
was when they made changes in the one with RW access.  They needed to 
unmount and remount all the RO clients to see the updates.  It was not 
graceful and we advised that they stop doing that.  But it did work for 
their needs.  They used it akin to a bit (~TB) CD-ROM.

> In a container environment the two containers aren't really isolated
> at the actual physical filesystem level since they share the kernel,

I think mount namespaces muddy this water.  Yes, it's the same kernel, 
but the containers don't have the same file systems exposed to the 
container.

> so I think you'd be fine but I'd really want to test or do some
> research before relying on it.

Yes, test.

But make sure you have a vague understanding of what's actually 
happening behind the scenes.  I find that tremendously helpful in 
knowing what can and can't be done, as well as why.

> In any case, the more typical solution is to just mount everything on
> the host and then bind-mount it into the container.  So, you could
> mount the nfs in /mnt and then bind-mount that into your container.
> There is really no performance hit and it should work fine without
> giving the container a bunch of capabilities.

I think there /is/ a performance hit.  It's just so /minimal/ that it's 
effectively non-existent.  Every additional line of code in the path 
that must be traversed does take CPU cycles.