Re: [gentoo-server] DoS Analysis and Prevemption

[Kerin Millar]

On 15/04/2013 16:07, Christian Parpart wrote:
> Hey all,
>
> we hit some nice traffic last night that took our main gateway down.
> Pacemaker was configured to failover to our second one, but that one
> died aswell.
>
> In a little post-analysis, I found the following in the logs:
>
> Apr 14 21:42:11 cesar1 kernel: [27613652.439846] BUG: soft lockup -
> CPU#4 stuck for 22s! [swapper/4:0]
> Apr 14 21:42:11 cesar1 kernel: [27613652.440319] Stack:
> Apr 14 21:42:11 cesar1 kernel: [27613652.440446] Call Trace:
> Apr 14 21:42:11 cesar1 kernel: [27613652.440595]  <IRQ>
> Apr 14 21:42:12 cesar1 kernel: [27613652.440828]  <EOI>
> Apr 14 21:42:12 cesar1 kernel: [27613652.440979] Code: c1 51 da 03 81 48
> c7 c2 4e da 03 81 e9 dd fe ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90
> 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 <89> c2
> Apr 14 21:42:12 cesar1 CRON[13599]: nss_ldap: could not connect to any
> LDAP server as cn=admin,dc=rz,dc=dawanda,dc=com - Can't contact LDAP server
> Apr 14 21:42:12 cesar1 CRON[13599]: nss_ldap: could not search LDAP
> server - Server is unavailable
> Apr 14 21:42:24 cesar1 crmd: [7287]: ERROR: process_lrm_event: LRM
> operation management-gateway-ip1_stop_0 (917) Timed Out (timeout=20000ms)
> Apr 14 21:42:48 cesar1 kernel: [27613688.611501] BUG: soft lockup -
> CPU#7 stuck for 22s! [named:32166]
> Apr 14 21:42:48 cesar1 kernel: [27613688.611914] Stack:
> Apr 14 21:42:48 cesar1 kernel: [27613688.612036] Call Trace:
> Apr 14 21:42:48 cesar1 kernel: [27613688.612200]  <IRQ>
> Apr 14 21:42:48 cesar1 kernel: [27613688.612408]  <EOI>
> Apr 14 21:42:48 cesar1 kernel: [27613688.612626] Code: c1 51 da 03 81 48
> c7 c2 4e da 03 81 e9 dd fe ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90
> 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 <89> c2
> Apr 14 21:42:55 cesar1 kernel: [27613695.946295] BUG: soft lockup -
> CPU#0 stuck for 21s! [ksoftirqd/0:3]
> Apr 14 21:42:55 cesar1 kernel: [27613695.946785] Stack:
> Apr 14 21:42:55 cesar1 kernel: [27613695.946917] Call Trace:
> Apr 14 21:42:55 cesar1 kernel: [27613695.947137] Code: c4 00 00 81 a8 44
> e0 ff ff ff 01 00 00 48 63 80 44 e0 ff ff a9 00 ff ff 07 74 36 65 48 8b
> 04 25 c8 c4 00 00 83 a8 44 e0 ff ff 01 <5d> c3
>
> We're using irqbalance to not only hit the first CPU for ethernet card
> hardware interrupts when traffic comes in (learned from last much more
> intensive DDoS).

To use irqbalance is wise. You could also try using receive packet 
steering [1] [2]:

#!/bin/bash
iface='eth*'
flow=16384
echo $flow > /proc/sys/net/core/rps_sock_flow_entries
queues=(/sys/class/net/${iface}/queues/rx-*)
for rx in "${queues[@]}"; do
         echo $(sed -e 's/0/f/g' < $rx/rps_cpus) > $rx/rps_cpus
         echo $flow > $rx/rps_flow_cnt
done

I have found this to be beneficial on systems running networking 
applications that are subject to a high load, but not for systems that 
are simply forwarding packets and processing them entirely in kernel space.

> However, since this not helped, I'd like to find out what else we can
> do. Our gateway has to do NAT and has a few other iptables rules it
> needs in order to run OpenStack behind,
> so I can't just drop it.
>
> Regarding the logs, I can see, that something caused the CPU cores to
> get stuck for a number of different processes.
> Has anyone ever encountered such error messages I quoted above or knows

I used to encounter them but they cleared up at some point during the 
3.4 (longterm) kernel series. If you also use the 3.4 series, I would 
advise upgrading if running < 3.4.51. If you are not using a longterm 
kernel, consider doing so unless there is a feature in a later kernel 
that you cannot do without. My experience of the later 'stable' kernels 
lately is that they have a tendency to introduce serious regressions.

> other things one might want to do in order to prevent hugh unsocialized
> incoming traffic from bringing a Linux node down?

If you can, talk with your upstream to see if there is a way in which 
such traffic can be throttled there.

Be sure to use good quality NICs. In particular, it should support 
multiqueue and adjustable interrupt coalescing (preferably on a dynamic 
basis). For what it's worth, I'm using Intel 82576 based cards for busy 
hosts. These support dynamic interrupt throttling. Even without such a 
feature, some cards will allow their behaviour to be altered via ethtool 
-C. Google will turn up a lot of information on this topic.

I should add that the stability of the driver is of paramount 
performance. Though my Intel cards have been solid, the igb driver 
bundled with the 3.4 kernel is not, which took me a long time to figure 
out. I now use a local ebuild to compile the igb driver from upstream. 
Not only did it improve performance, but it resolved all stability 
issues that I had experienced up until then.

In the event that you are also using the igb driver, ensure that it is 
configured optimally for multiqueue. Here's an example for the upstream 
driver (my NIC has 4 ports, each with 8 queues):

# cat /etc/modprobe.d/igb.conf
options igb RSS=8,8,8,8

Enable I/OAT if your hardware supports it. Some hardware will support it 
but fail to expose a BIOS option to enable it, in which case you can try 
using dca_force [3] (YMMV). Similarly, make use of x2APIC if supported, 
but do not make use of the IOMMU provided by Intel as of Nehalem (boot 
with intel_iommu=off if in doubt).

Consider fine-tuning sysctl.conf, especially those pertaining to buffer 
sizes/limits. I would consider this essential if operating at gigabit 
speeds or higher. Examples are widespread, such as in section 3.1 of the 
Mellanox performance tuning guide [4].

--Kerin

[1] https://lwn.net/Articles/361440/
[2] http://thread.gmane.org/gmane.linux.network/179883/focus=179976
[3] https://github.com/ice799/dca_force
[4] 
http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters_rev_1_0.pdf