| 1 |
On Sunday, 8 October 2017 12:20:00 BST Hubert Hauser wrote: |
| 2 |
> On 07/10/17 18:50, Mick wrote: |
| 3 |
> > On Saturday, 7 October 2017 17:23:33 BST Hubert Hauser wrote: |
| 4 |
> >> I am using Gentoo as Host OS for Docker containers. |
| 5 |
> >> I have compiled |
| 6 |
> >> kernel using instructions on page |
| 7 |
> >> https://wiki.gentoo.org/wiki/Docker#Kernel and I have installed Docker |
| 8 |
> >> from Gentoo repository. |
| 9 |
> >> |
| 10 |
> >> Host system informations: |
| 11 |
> >> pecan@tux ~ $ uname -a |
| 12 |
> >> Linux tux 4.12.12-gentoo #8 SMP Sat Oct 7 13:58:47 CEST 2017 x86_64 |
| 13 |
> >> |
| 14 |
> >> Intel(R) Core(TM) i5-6300HQ CPU @ 2.30GHz GenuineIntel GNU/Linux |
| 15 |
> >> |
| 16 |
> >> Docker version: |
| 17 |
> >> pecan@tux ~ $ docker version |
| 18 |
> >> |
| 19 |
> >> Client: |
| 20 |
> >> Version: 17.03.2-ce |
| 21 |
> >> API version: 1.27 |
| 22 |
> >> Go version: go1.9.1 |
| 23 |
> >> Git commit: f5ec1e2 |
| 24 |
> >> Built: Sat Oct 7 14:50:59 2017 |
| 25 |
> >> OS/Arch: linux/amd64 |
| 26 |
> >> |
| 27 |
> >> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. |
| 28 |
> >> |
| 29 |
> >> Is the docker daemon running? |
| 30 |
> >> |
| 31 |
> >> Look at "Cannot connect to the Docker daemon at |
| 32 |
> >> unix:///var/run/docker.sock. Is the docker daemon running?". The same |
| 33 |
> >> |
| 34 |
> >> message appears if I try get docker system-wide informations: |
| 35 |
> >> pecan@tux ~ $ docker info |
| 36 |
> >> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. |
| 37 |
> >> |
| 38 |
> >> Is the docker daemon running? |
| 39 |
> > |
| 40 |
> > It seems you have not yet started docker. |
| 41 |
> |
| 42 |
> I can't start it because failure occurs when starting Docker daemon. |
| 43 |
> |
| 44 |
> On 07/10/17 18:50, Mick wrote: |
| 45 |
> >> The same error appears if I try run the same command as sudo, so this |
| 46 |
> >> error applies to daemon. I tried to check if there a mistake in Docker |
| 47 |
> >> daemon privileges. |
| 48 |
> >> |
| 49 |
> >> pecan@tux ~ $ sudo docker info |
| 50 |
> >> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. |
| 51 |
> >> |
| 52 |
> >> Is the docker daemon running? |
| 53 |
> >> |
| 54 |
> >> Based on the message I am able to say that maybe Docker daemon not |
| 55 |
> >> |
| 56 |
> >> running. I checked daemon status to make sure: |
| 57 |
> >> pecan@tux ~ $ sudo service docker status |
| 58 |
> >> |
| 59 |
> >> * status: crashed |
| 60 |
> > |
| 61 |
> > Did you try starting it from the CLI? Any useful messages there? |
| 62 |
> |
| 63 |
> Yes. |
| 64 |
> |
| 65 |
> The result of command sudo dockerd: |
| 66 |
> https://paste.pound-python.org/show/AyjLQEBGABpjo57E9WMa/ |
| 67 |
> and of sudo dockerd -s overlay2: |
| 68 |
> https://paste.pound-python.org/show/ulyqb7D363I51JqkvrEk/. |
| 69 |
> |
| 70 |
> You can also look at attachments dockerd.txt (sudo dockerd) and |
| 71 |
> dockerd-overlay2.txt (sudo dockerd -s overlay2). |
| 72 |
> |
| 73 |
> I must run dockerd without connecting into VPN. |
| 74 |
> |
| 75 |
> I've reset iptables and ip6tables rules to defaults: |
| 76 |
> |
| 77 |
> pecan@tux ~ $ sudo iptables -L |
| 78 |
> Chain INPUT (policy ACCEPT) |
| 79 |
> target prot opt source destination |
| 80 |
> |
| 81 |
> Chain FORWARD (policy ACCEPT) |
| 82 |
> target prot opt source destination |
| 83 |
> DOCKER-ISOLATION all -- anywhere anywhere |
| 84 |
> DOCKER all -- anywhere anywhere |
| 85 |
> ACCEPT all -- anywhere anywhere ctstate |
| 86 |
> RELATED,ESTABLISHED |
| 87 |
> ACCEPT all -- anywhere anywhere |
| 88 |
> ACCEPT all -- anywhere anywhere |
| 89 |
> |
| 90 |
> Chain OUTPUT (policy ACCEPT) |
| 91 |
> target prot opt source destination |
| 92 |
> |
| 93 |
> Chain DOCKER (1 references) |
| 94 |
> target prot opt source destination |
| 95 |
> |
| 96 |
> Chain DOCKER-ISOLATION (1 references) |
| 97 |
> target prot opt source destination |
| 98 |
> RETURN all -- anywhere anywhere |
| 99 |
> |
| 100 |
> (tensorflow) pecan@tux ~ $ sudo ip6tables -L |
| 101 |
> Chain INPUT (policy ACCEPT) |
| 102 |
> target prot opt source destination |
| 103 |
> |
| 104 |
> Chain FORWARD (policy ACCEPT) |
| 105 |
> target prot opt source destination |
| 106 |
> |
| 107 |
> Chain OUTPUT (policy ACCEPT) |
| 108 |
> target prot opt source destination |
| 109 |
> |
| 110 |
> Here's my Docker logs: |
| 111 |
> |
| 112 |
> (tensorflow) pecan@tux ~ $ sudo cat /var/log/docker-err.log |
| 113 |
> time="2017-10-07T14:54:24.561887891+02:00" level=info |
| 114 |
> msg="libcontainerd: new containerd process, pid: 994" |
| 115 |
> Error starting daemon: error initializing graphdriver: prerequisites for |
| 116 |
> driver not satisfied (wrong filesystem?) |
| 117 |
> time="2017-10-07T14:56:44.682987498+02:00" level=info |
| 118 |
> msg="libcontainerd: new containerd process, pid: 4983" |
| 119 |
> Error starting daemon: error initializing graphdriver: prerequisites for |
| 120 |
> driver not satisfied (wrong filesystem?) |
| 121 |
> time="2017-10-07T17:00:16.938724808+02:00" level=info |
| 122 |
> msg="libcontainerd: new containerd process, pid: 7254" |
| 123 |
> Error starting daemon: error initializing graphdriver: prerequisites for |
| 124 |
> driver not satisfied (wrong filesystem?) |
| 125 |
> time="2017-10-07T17:03:58.151814419+02:00" level=info |
| 126 |
> msg="libcontainerd: new containerd process, pid: 7857" |
| 127 |
> Error starting daemon: error initializing graphdriver: prerequisites for |
| 128 |
> driver not satisfied (wrong filesystem?) |
| 129 |
> time="2017-10-07T17:05:09.492506131+02:00" level=info |
| 130 |
> msg="libcontainerd: new containerd process, pid: 7964" |
| 131 |
> Error starting daemon: error initializing graphdriver: prerequisites for |
| 132 |
> driver not satisfied (wrong filesystem?) |
| 133 |
> time="2017-10-07T17:06:27.126674008+02:00" level=info |
| 134 |
> msg="libcontainerd: new containerd process, pid: 8202" |
| 135 |
> Error starting daemon: error initializing graphdriver: prerequisites for |
| 136 |
> driver not satisfied (wrong filesystem?) |
| 137 |
> time="2017-10-07T17:09:47.482580356+02:00" level=info |
| 138 |
> msg="libcontainerd: new containerd process, pid: 8730" |
| 139 |
> Error starting daemon: error initializing graphdriver: prerequisites for |
| 140 |
> driver not satisfied (wrong filesystem?) |
| 141 |
> time="2017-10-07T21:16:26.088461863+02:00" level=info |
| 142 |
> msg="libcontainerd: new containerd process, pid: 6414" |
| 143 |
> Error starting daemon: error initializing graphdriver: prerequisites for |
| 144 |
> driver not satisfied (wrong filesystem?) |
| 145 |
> time="2017-10-08T10:20:56.011379547+02:00" level=debug msg="docker group |
| 146 |
> found. gid: 987" |
| 147 |
> time="2017-10-08T10:20:56.023468730+02:00" level=debug msg="Listener |
| 148 |
> created for HTTP on unix (/var/run/docker.sock)" |
| 149 |
> time="2017-10-08T10:20:56.023686388+02:00" level=info |
| 150 |
> msg="libcontainerd: new containerd process, pid: 26284" |
| 151 |
> time="2017-10-08T10:20:56.030713286+02:00" level=debug msg="containerd: |
| 152 |
> read past events" count=0 |
| 153 |
> time="2017-10-08T10:20:56.030808207+02:00" level=debug msg="containerd: |
| 154 |
> supervisor running" cpus=4 memory=7849 runtime=docker-runc |
| 155 |
> runtimeArgs=[] stateDir="/var/run/docker/libcontainerd/containerd" |
| 156 |
> time="2017-10-08T10:20:56.030871453+02:00" level=debug msg="containerd: |
| 157 |
> grpc api on /var/run/docker/libcontainerd/docker-containerd.sock" |
| 158 |
> time="2017-10-08T10:20:56.524998596+02:00" level=debug |
| 159 |
> msg="libcontainerd: containerd health check returned error: rpc error: |
| 160 |
> code = 14 desc = grpc: the connection is unavailable" |
| 161 |
> time="2017-10-08T10:20:57.035916104+02:00" level=debug msg="Using |
| 162 |
> default logging driver json-file" |
| 163 |
> time="2017-10-08T10:20:57.035960712+02:00" level=debug msg="Golang's |
| 164 |
> threads limit set to 56430" |
| 165 |
> time="2017-10-08T10:20:57.036012700+02:00" level=debug |
| 166 |
> msg="[graphdriver] trying provided driver: btrfs" |
| 167 |
> time="2017-10-08T10:20:57.036027034+02:00" level=debug msg="Cleaning up |
| 168 |
> old mountid : start." |
| 169 |
> Error starting daemon: error initializing graphdriver: prerequisites for |
| 170 |
> driver not satisfied (wrong filesystem?) |
| 171 |
> time="2017-10-08T10:21:05.597682241+02:00" level=debug msg="docker group |
| 172 |
> found. gid: 987" |
| 173 |
> time="2017-10-08T10:21:05.597724031+02:00" level=debug msg="Listener |
| 174 |
> created for HTTP on unix (/var/run/docker.sock)" |
| 175 |
> time="2017-10-08T10:21:05.597910790+02:00" level=info |
| 176 |
> msg="libcontainerd: new containerd process, pid: 26392" |
| 177 |
> time="2017-10-08T10:21:05.606126357+02:00" level=debug msg="containerd: |
| 178 |
> read past events" count=0 |
| 179 |
> time="2017-10-08T10:21:05.606219342+02:00" level=debug msg="containerd: |
| 180 |
> supervisor running" cpus=4 memory=7849 runtime=docker-runc |
| 181 |
> runtimeArgs=[] stateDir="/var/run/docker/libcontainerd/containerd" |
| 182 |
> time="2017-10-08T10:21:05.606245984+02:00" level=debug msg="containerd: |
| 183 |
> grpc api on /var/run/docker/libcontainerd/docker-containerd.sock" |
| 184 |
> time="2017-10-08T10:21:06.114930279+02:00" level=debug |
| 185 |
> msg="libcontainerd: containerd health check returned error: rpc error: |
| 186 |
> code = 14 desc = grpc: the connection is unavailable" |
| 187 |
> time="2017-10-08T10:21:06.614858989+02:00" level=debug |
| 188 |
> msg="libcontainerd: containerd health check returned error: rpc error: |
| 189 |
> code = 14 desc = grpc: the connection is unavailable" |
| 190 |
> time="2017-10-08T10:21:06.645208467+02:00" level=debug msg="Using |
| 191 |
> default logging driver json-file" |
| 192 |
> time="2017-10-08T10:21:06.645236784+02:00" level=debug msg="Golang's |
| 193 |
> threads limit set to 56430" |
| 194 |
> time="2017-10-08T10:21:06.645311511+02:00" level=debug |
| 195 |
> msg="[graphdriver] trying provided driver: btrfs" |
| 196 |
> time="2017-10-08T10:21:06.645324876+02:00" level=debug msg="Cleaning up |
| 197 |
> old mountid : start." |
| 198 |
> Error starting daemon: error initializing graphdriver: prerequisites for |
| 199 |
> driver not satisfied (wrong filesystem?) |
| 200 |
> (tensorflow) pecan@tux ~ $ sudo cat /var/log/docker.log |
| 201 |
> time="2017-10-07T14:52:13.178261811+02:00" level=info |
| 202 |
> msg="libcontainerd: new containerd process, pid: 32311" |
| 203 |
> time="2017-10-07T14:52:14.434232306+02:00" level=info msg="Graph |
| 204 |
> migration to content-addressability took 0.00 seconds" |
| 205 |
> time="2017-10-07T14:52:14.434413425+02:00" level=warning msg="Your |
| 206 |
> kernel does not support cgroup blkio weight" |
| 207 |
> time="2017-10-07T14:52:14.434423960+02:00" level=warning msg="Your |
| 208 |
> kernel does not support cgroup blkio weight_device" |
| 209 |
> time="2017-10-07T14:52:14.434759986+02:00" level=info msg="Loading |
| 210 |
> containers: start." |
| 211 |
> time="2017-10-07T14:52:14.437180876+02:00" level=info msg="Firewalld |
| 212 |
> running: false" |
| 213 |
> Error starting daemon: Error initializing network controller: list |
| 214 |
> bridge addresses failed: no available network |
| 215 |
> (tensorflow) pecan@tux ~ $ sudo cat /var/log/docker-out.log |
| 216 |
> (tensorflow) pecan@tux ~ $ |
| 217 |
> |
| 218 |
> Docker daemon options: |
| 219 |
> |
| 220 |
> (tensorflow) pecan@tux ~ $ sudo cat /etc/docker/daemon.json |
| 221 |
> { |
| 222 |
> "debug": true |
| 223 |
> } |
| 224 |
> |
| 225 |
> On 07/10/17 18:50, Mick wrote: |
| 226 |
> >> Docker daemon is crashed. To see the reason, I looked at the logs: |
| 227 |
> >> pecan@tux ~ $ cat /var/log/docker.log |
| 228 |
> >> time="2017-10-07T14:52:13.178261811+02:00" level=info |
| 229 |
> >> |
| 230 |
> >> msg="libcontainerd: new containerd process, pid: 32311" |
| 231 |
> >> |
| 232 |
> >> time="2017-10-07T14:52:14.434232306+02:00" level=info msg="Graph |
| 233 |
> >> |
| 234 |
> >> migration to content-addressability took 0.00 seconds" |
| 235 |
> >> |
| 236 |
> >> time="2017-10-07T14:52:14.434413425+02:00" level=warning msg="Your |
| 237 |
> >> |
| 238 |
> >> kernel does not support cgroup blkio weight" |
| 239 |
> > |
| 240 |
> > OK, start from checking your kernel has all the necessary modules compiled |
| 241 |
> > in, rebuild it and reboot. |
| 242 |
> |
| 243 |
> Look here: |
| 244 |
> |
| 245 |
> pecan@tux ~ $ sudo ~/check-config.sh |
| 246 |
> info: reading kernel config from /proc/config.gz ... |
| 247 |
> |
| 248 |
> Generally Necessary: |
| 249 |
> - cgroup hierarchy: properly mounted [/sys/fs/cgroup] |
| 250 |
> - CONFIG_NAMESPACES: enabled |
| 251 |
> - CONFIG_NET_NS: enabled |
| 252 |
> - CONFIG_PID_NS: enabled |
| 253 |
> - CONFIG_IPC_NS: enabled |
| 254 |
> - CONFIG_UTS_NS: enabled |
| 255 |
> - CONFIG_CGROUPS: enabled |
| 256 |
> - CONFIG_CGROUP_CPUACCT: enabled |
| 257 |
> - CONFIG_CGROUP_DEVICE: enabled |
| 258 |
> - CONFIG_CGROUP_FREEZER: enabled |
| 259 |
> - CONFIG_CGROUP_SCHED: enabled |
| 260 |
> - CONFIG_CPUSETS: enabled |
| 261 |
> - CONFIG_MEMCG: enabled |
| 262 |
> - CONFIG_KEYS: enabled |
| 263 |
> - CONFIG_VETH: enabled |
| 264 |
> - CONFIG_BRIDGE: enabled |
| 265 |
> - CONFIG_BRIDGE_NETFILTER: enabled |
| 266 |
> - CONFIG_NF_NAT_IPV4: enabled |
| 267 |
> - CONFIG_IP_NF_FILTER: enabled |
| 268 |
> - CONFIG_IP_NF_TARGET_MASQUERADE: enabled |
| 269 |
> - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled |
| 270 |
> - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled |
| 271 |
> - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module) |
| 272 |
> - CONFIG_IP_NF_NAT: enabled |
| 273 |
> - CONFIG_NF_NAT: enabled |
| 274 |
> - CONFIG_NF_NAT_NEEDED: enabled |
| 275 |
> - CONFIG_POSIX_MQUEUE: enabled |
| 276 |
> |
| 277 |
> Optional Features: |
| 278 |
> - CONFIG_USER_NS: enabled |
| 279 |
> - CONFIG_SECCOMP: enabled |
| 280 |
> - CONFIG_CGROUP_PIDS: enabled |
| 281 |
> - CONFIG_MEMCG_SWAP: enabled |
| 282 |
> - CONFIG_MEMCG_SWAP_ENABLED: enabled |
| 283 |
> (cgroup swap accounting is currently enabled) |
| 284 |
> - CONFIG_LEGACY_VSYSCALL_EMULATE: enabled |
| 285 |
> - CONFIG_BLK_CGROUP: enabled |
| 286 |
> - CONFIG_BLK_DEV_THROTTLING: enabled |
| 287 |
> - CONFIG_IOSCHED_CFQ: enabled (as module) |
| 288 |
> - CONFIG_CFQ_GROUP_IOSCHED: missing |
| 289 |
> - CONFIG_CGROUP_PERF: enabled |
| 290 |
> - CONFIG_CGROUP_HUGETLB: missing |
| 291 |
> - CONFIG_NET_CLS_CGROUP: enabled |
| 292 |
> - CONFIG_CGROUP_NET_PRIO: enabled |
| 293 |
> - CONFIG_CFS_BANDWIDTH: enabled |
| 294 |
> - CONFIG_FAIR_GROUP_SCHED: enabled |
| 295 |
> - CONFIG_RT_GROUP_SCHED: enabled |
| 296 |
> - CONFIG_IP_VS: enabled (as module) |
| 297 |
> - CONFIG_IP_VS_NFCT: enabled |
| 298 |
> - CONFIG_IP_VS_RR: enabled (as module) |
| 299 |
> - CONFIG_EXT3_FS: enabled |
| 300 |
> - CONFIG_EXT3_FS_XATTR: missing |
| 301 |
> - CONFIG_EXT3_FS_POSIX_ACL: enabled |
| 302 |
> - CONFIG_EXT3_FS_SECURITY: enabled |
| 303 |
> (enable these ext3 configs if you are using ext3 as backing filesystem) |
| 304 |
> - CONFIG_EXT4_FS: enabled |
| 305 |
> - CONFIG_EXT4_FS_POSIX_ACL: enabled |
| 306 |
> - CONFIG_EXT4_FS_SECURITY: enabled |
| 307 |
> - Network Drivers: |
| 308 |
> - "overlay": |
| 309 |
> - CONFIG_VXLAN: enabled (as module) |
| 310 |
> Optional (for encrypted networks): |
| 311 |
> - CONFIG_CRYPTO: enabled |
| 312 |
> - CONFIG_CRYPTO_AEAD: enabled |
| 313 |
> - CONFIG_CRYPTO_GCM: enabled |
| 314 |
> - CONFIG_CRYPTO_SEQIV: enabled |
| 315 |
> - CONFIG_CRYPTO_GHASH: enabled |
| 316 |
> - CONFIG_XFRM: enabled |
| 317 |
> - CONFIG_XFRM_USER: enabled (as module) |
| 318 |
> - CONFIG_XFRM_ALGO: enabled (as module) |
| 319 |
> - CONFIG_INET_ESP: enabled (as module) |
| 320 |
> - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module) |
| 321 |
> - "ipvlan": |
| 322 |
> - CONFIG_IPVLAN: enabled (as module) |
| 323 |
> - "macvlan": |
| 324 |
> - CONFIG_MACVLAN: enabled (as module) |
| 325 |
> - CONFIG_DUMMY: enabled (as module) |
| 326 |
> - "ftp,tftp client in container": |
| 327 |
> - CONFIG_NF_NAT_FTP: enabled |
| 328 |
> - CONFIG_NF_CONNTRACK_FTP: enabled |
| 329 |
> - CONFIG_NF_NAT_TFTP: enabled |
| 330 |
> - CONFIG_NF_CONNTRACK_TFTP: enabled |
| 331 |
> - Storage Drivers: |
| 332 |
> - "aufs": |
| 333 |
> - CONFIG_AUFS_FS: missing |
| 334 |
> - "btrfs": |
| 335 |
> - CONFIG_BTRFS_FS: enabled |
| 336 |
> - CONFIG_BTRFS_FS_POSIX_ACL: enabled |
| 337 |
> - "devicemapper": |
| 338 |
> - CONFIG_BLK_DEV_DM: enabled |
| 339 |
> - CONFIG_DM_THIN_PROVISIONING: enabled |
| 340 |
> - "overlay": |
| 341 |
> - CONFIG_OVERLAY_FS: enabled |
| 342 |
> - "zfs": |
| 343 |
> - /dev/zfs: missing |
| 344 |
> - zfs command: missing |
| 345 |
> - zpool command: missing |
| 346 |
> |
| 347 |
> Limits: |
| 348 |
> - /proc/sys/kernel/keys/root_maxkeys: 1000000 |
| 349 |
> |
| 350 |
> My kernel config: |
| 351 |
> https://paste.pound-python.org/show/bFHNp4jrHE8kKzPMtNGq/ (in attachment |
| 352 |
> file kernel-config.txt, sudo zcat /proc/config.gz). |
| 353 |
> |
| 354 |
> -- |
| 355 |
> Best regards, |
| 356 |
> Hubert Hauser. |
| 357 |
|
| 358 |
|
| 359 |
I have not trimmed your info above, in case someone more knowledgeable in |
| 360 |
running docker can advise. |
| 361 |
|
| 362 |
From what I see above you are running btrfs. It may be worth compiling in |
| 363 |
your kernel this module you have left out, because I've read somewhere it |
| 364 |
prevents fs corruption (Rich seems to know a lot about BTRFS, so I leave it to |
| 365 |
him to confirm its usefulness): |
| 366 |
|
| 367 |
# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set |
| 368 |
|
| 369 |
|
| 370 |
Two warnings in /var/log/docker-err.log attracted my attention: |
| 371 |
|
| 372 |
"Your kernel does not support cgroup blkio weight" |
| 373 |
|
| 374 |
Try settings fair queue scheduling for your storage and check it is being |
| 375 |
used. |
| 376 |
|
| 377 |
The second thing is caused by the filesystem: |
| 378 |
|
| 379 |
... prerequisites for driver not satisfied (wrong filesystem?) |
| 380 |
|
| 381 |
|
| 382 |
You could try starting docker with options '-d -g' which apparently works for |
| 383 |
booting btrfs as advised here: |
| 384 |
|
| 385 |
https://stackoverflow.com/questions/30669359/docker-with-btrfs-ubuntu |
| 386 |
|
| 387 |
HTH. |
| 388 |
-- |
| 389 |
Regards, |
| 390 |
Mick |
Archives