Re: [gentoo-user] Networking trouble

[hw]

J. Roeleveld wrote:
> On Thursday, October 15, 2015 03:30:01 PM hw wrote:
>> Hi,
>>
>> I have a xen host with some HV guests which becomes unreachable via
>> the network after apparently random amount of times.  I have already
>> switched the network card to see if that would make a difference,
>> and with the card currently installed, it worked fine for over 20 days
>> until it become unreachable again.  Before switching the network card,
>> it would run a week or two before becoming unreachable.  The previous
>> card was the on-board BCM5764M which uses the tg3 driver.
>>
>> There are messages like this in the log file:
>>
>>
>> Oct 14 20:58:02 moonflo kernel: ------------[ cut here ]------------
>> Oct 14 20:58:02 moonflo kernel: WARNING: CPU: 10 PID: 0 at
>> net/sched/sch_generic.c:303 dev_watchdog+0x259/0x270() Oct 14 20:58:02
>> moonflo kernel: NETDEV WATCHDOG: enp55s4 (r8169): transmit queue 0 timed
>> out Oct 14 20:58:02 moonflo kernel: Modules linked in: arc4 ecb md4 hmac
>> nls_utf8 cifs fscache xt_physdev br_netfilter iptable_filter ip_tables
>> xen_pciback xen_gntalloc xen_gntdev bridge stp llc zfs(PO) nouveau
>> snd_hda_codec_realtek snd_hda_codec_generic zunicode(PO) zavl(PO)
>> zcommon(PO) znvpair(PO) spl(O) zlib_deflate video backlight drm_kms_helper
>> ttm snd_hda_intel snd_hda_controller snd_hda_codec snd_pcm snd_timer snd
>> soundcore r8169 mii xts aesni_intel glue_helper lrw gf128mul ablk_helper
>> cryptd aes_x86_64 sha256_generic hid_generic usbhid uhci_hcd usb_storage
>> ehci_pci ehci_hcd usbcore usb_common Oct 14 20:58:02 moonflo kernel: CPU:
>> 10 PID: 0 Comm: swapper/10 Tainted: P           O    4.0.5-gentoo #3 Oct 14
>> 20:58:02 moonflo kernel: Hardware name: Hewlett-Packard HP Z800
>> Workstation/0AECh, BIOS 786G5 v03.57 07/15/2013 Oct 14 20:58:02 moonflo
>> kernel:  ffffffff8175a77d ffff880124d43d98 ffffffff814da8d8
>> 0000000000000001 Oct 14 20:58:02 moonflo kernel:  ffff880124d43de8
>> ffff880124d43dd8 ffffffff81088850 ffff880124d43dd8 Oct 14 20:58:02 moonflo
>> kernel:  0000000000000000 ffff8800d45f2000 0000000000000001
>> ffff8800d5294880 Oct 14 20:58:02 moonflo kernel: Call Trace:
>> Oct 14 20:58:02 moonflo kernel:  <IRQ>  [<ffffffff814da8d8>]
>> dump_stack+0x45/0x57 Oct 14 20:58:02 moonflo kernel:  [<ffffffff81088850>]
>> warn_slowpath_common+0x80/0xc0 Oct 14 20:58:02 moonflo kernel:
>> [<ffffffff810888d1>] warn_slowpath_fmt+0x41/0x50 Oct 14 20:58:02 moonflo
>> kernel:  [<ffffffff812b31c5>] ? add_interrupt_randomness+0x35/0x1e0 Oct 14
>> 20:58:02 moonflo kernel:  [<ffffffff8145b819>] dev_watchdog+0x259/0x270 Oct
>> 14 20:58:02 moonflo kernel:  [<ffffffff8145b5c0>] ?
>> dev_graft_qdisc+0x80/0x80 Oct 14 20:58:02 moonflo kernel:
>> [<ffffffff8145b5c0>] ? dev_graft_qdisc+0x80/0x80 Oct 14 20:58:02 moonflo
>> kernel:  [<ffffffff810d4047>] call_timer_fn.isra.30+0x17/0x70 Oct 14
>> 20:58:02 moonflo kernel:  [<ffffffff810d42a6>]
>> run_timer_softirq+0x176/0x2b0 Oct 14 20:58:02 moonflo kernel:
>> [<ffffffff8108bd0a>] __do_softirq+0xda/0x1f0 Oct 14 20:58:02 moonflo
>> kernel:  [<ffffffff8108c04e>] irq_exit+0x7e/0xa0 Oct 14 20:58:02 moonflo
>> kernel:  [<ffffffff8130e075>] xen_evtchn_do_upcall+0x35/0x50 Oct 14
>> 20:58:02 moonflo kernel:  [<ffffffff814e1e8e>]
>> xen_do_hypervisor_callback+0x1e/0x40 Oct 14 20:58:02 moonflo kernel:  <EOI>
>>   [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 Oct 14 20:58:02
>> moonflo kernel:  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 Oct
>> 14 20:58:02 moonflo kernel:  [<ffffffff810459e0>] ? xen_safe_halt+0x10/0x20
>> Oct 14 20:58:02 moonflo kernel:  [<ffffffff81053979>] ?
>> default_idle+0x9/0x10 Oct 14 20:58:02 moonflo kernel:  [<ffffffff810542da>]
>> ? arch_cpu_idle+0xa/0x10 Oct 14 20:58:02 moonflo kernel:
>> [<ffffffff810bd170>] ? cpu_startup_entry+0x190/0x2f0 Oct 14 20:58:02
>> moonflo kernel:  [<ffffffff81047cd5>] ? cpu_bringup_and_idle+0x25/0x40 Oct
>> 14 20:58:02 moonflo kernel: ---[ end trace 98d961bae351244d ]--- Oct 14
>> 20:58:02 moonflo kernel: r8169 0000:37:04.0 enp55s4: link up
>>
>>
>> After that, there are lots of messages about the link being up, one message
>> every 12 seconds.  When you unplug the network cable, you get a message that
>> the link is down, and no message when you plug it in again.
>>
>> I was hoping that switching the network card (to one that uses a different
>> driver) might solve the problem, and it did not.  Now I can only guess that
>> the network card goes to sleep and sometimes cannot be woken up again.
>>
>> I tried to reduce the connection speed to 100Mbit and found that accessing
>> the VMs (via RDP) becomes too slow to use them.  So I disabled the power
>> management of the network card (through sysfs) and will have to see if the
>> problem persists.
>>
>> We'll be getting decent network cards in a couple days, but since the
>> problem doesn't seem to be related to a particular card/model/manufacturer,
>> that might not fix it, either.
>>
>> This problem seems to only occur on machines that operate as a xen server.
>> Other machines, identical Z800s, not running xen, run just fine.
>>
>> What would you suggest?
>
> More info required:
>
> - Which version of Xen

4.5.1

Installed versions:  4.5.1^t(02:44:35 PM 07/14/2015)(-custom-cflags -debug -efi -flask -xsm)

> - Does this only occur with HVM guests?

The host has been running only HVM guests every time it happend.
It was running a PV guest in between (which I had to shut down
because other VMs were migrated, requiring the RAM).

> - Which network-driver are you using inside the guest

r8169, compiled as a module

Same happened with the tg3 driver when the on-board cards were used.
The tg3 driver is completely disabled in the kernel config, i. e.
not even compiled as a module.

> - Can you connect to the "local" console of the guest?

Yes, the host seems to be running fine except for having no network
connectivity.  There's a keyboard and monitor physically connected to
it with which you can log in and do stuff.

You get no answer when you ping the host while it is unreachable.

> - If yes, does it still have no connectivity?

It has been restarted this morning when it was found to be unreachable.

> I saw the same on my lab machine, which was related to:
> - Not using correct drivers inside HVM guests

There are Windoze 7 guests running that have PV drivers installed.
One of those has formerly been running on a VMware host and was
migrated on Tuesday.  I deinstalled the VMware tools from it.

Since Monday, a HVM Linux system (a modified 32-bit Debian) has also
been migrated from the VMware host to this one.  I don't know if it
has VMware tools installed (I guess it does because it could be shut
down via VMware) and how those might react now.  It's working, and I
don't want to touch it.

However, the problem already occured before this migration, when the
on-board cards were still used.

> - Switch hardware not keeping the MAC/IP/Port lists long enough

What might be the reason for the lists becoming too short?  Too many
devices connected to the network?

The host has been connected to two different switches and showed the
problem.  Previously, that was an 8-port 1Gb switch, now it's a 24-port
1Gb switch.  However, the 8-port switch is also connected to the 24-port
switch the host is now connected to.  (The 24-port switch connects it
"directly" to the rest of the network.)