Re: [gentoo-dev] Re: An example overlayfs sandbox test

[Rich Freeman]

On Sun, Sep 24, 2017 at 2:11 PM, Martin Vaeth <martin@×××××.de> wrote:
> Rich Freeman <rich0@g.o> wrote:
>> On Sun, Sep 24, 2017 at 4:24 AM, Martin Vaeth <martin@×××××.de> wrote:
>>> Tim Harder <radhermit@g.o> wrote:
>>>
>>> It is the big advantage of overlay that it is implemented in
>>> kernel and does not involve any time-consuming checks during
>>> normal file operations.
>>
>> Why would you expect containers to behave any differently?
>
> For overlay, there is only one directory to be checked in
> addition for every file access.
>
> For containers, at least a dozens of binds are minimally required
> (/usr /proc /sys /dev ...).

I wouldn't be surprised if it works with a single bind mount with
/proc and /dev and so on mounted on top of that.  You really don't
want to be passing these directories through to the host filesystem
anyway.

>
>> Now, I am concerned about the time to create the container, if we're
>> going to specify individual files, but the same would be true of an
>> overlay. [...]
>> to populate an overlayfs with just that specific list of files.
>
> No. For overlay you need only one mount (not even a bind)
> and only one directory traversal at the end to check for
> violations.

You say "not even a bind" as if that is a benefit.  I suspect bind
mounts operate more quickly than an overlayfs if anything.

> The nice thing is that this is practically independent of
> the number or structure of directories/files you want to protect,
> i.e. it scales perfectly well.
> For the more fine-grained approach, you just delete the files
> you do not want to have in the beginning. Not sure, how quick this
> can be done, but once it is done, the slowdown when running the
> sandbox is independent of the number of deleted files (because
> here certainly only one hash lookup is required).

Honestly, you can't really claim that overlayfs is superior to bind
mounts when it comes to access times without actually looking into how
fast bind mounts actually operate.  I'd have to read up on the kernel
VFS myself but people run hosts with lots of containers all the time
and they usually contain a ton of mountpoints.  The kernel obviously
has an efficient way to figure out what filesystem a path is actually
on, and it actually has to work this out even if you're using
overlayfs, since the kernel has to first figure out that the path is
even on the overlayfs.

It is possible that bind mount performance is inferior when you've
removed all but a thousand files from your overlayfs, and it is
possible that overlayfs performance is inferior.

>
>> If you just replicate the current sandbox
>> functionality then setup time is tiny
>
> I am not so much concerned about the setup time but more about the
> delay caused for file operations once the sandbox is set up.
> Perhaps even a dozen bind directories already give a considerable
> slowdown...
>

I run builds on Gentoo containers all the time and the host is
juggling dozens of bind mounts already.  Before I started using
containers I'd use bind mounts fairly often on monolithic hosts.  I
certainly haven't noticed any overhead.  There are certainly people
running FAR more containers per host than I am.  I wouldn't be
concerned with a couple of bind mounts. I have a ton of zfs
mountpoints as well and no issues.  (Bind mounts shouldn't have any
more cost than any other type of mount, and probably less.)

I wouldn't assume that thousands of bind mounts would have zero impact
without testing it, but I also have no reason to be concerned.

-- 
Rich