| 1 |
Hi! |
| 2 |
|
| 3 |
Today I found server nearly unresponsible (loadavg around 30, ssh type speed |
| 4 |
around few chars per second). It looks like nearly all processes (very |
| 5 |
different ones) eat each 3-5% CPU, with top's report about 95% CPU spend |
| 6 |
in "system" (i.e. not "user" or "wait"). At a glance it looks like kernel |
| 7 |
issue, so I checked kernel error log and found this, reported few hours ago: |
| 8 |
|
| 9 |
2010-05-12_03:51:29.90675 kern.err: PAX: refcount overflow detected in: fastcgi:32201, uid/euid: 1067/1067 |
| 10 |
2010-05-12_03:51:29.93807 kern.err: PAX: refcount overflow occured at: iret_exc+0x1d3e/0x4565 |
| 11 |
2010-05-12_03:51:29.93813 kern.warn: |
| 12 |
2010-05-12_03:51:29.94129 kern.warn: Pid: 32201, comm: fastcgi Tainted: G W (2.6.28-hardened-r9 #1) ProLiant DL140 G3 |
| 13 |
2010-05-12_03:51:29.94137 kern.warn: EIP: 0060:[<c06d95ee>] EFLAGS: 00000a96 CPU: 1 |
| 14 |
2010-05-12_03:51:29.94140 kern.warn: EIP is at iret_exc+0x1d3e/0x4565 |
| 15 |
2010-05-12_03:51:29.94143 kern.warn: EAX: 00000004 EBX: f756b080 ECX: df037a54 EDX: 00000070 |
| 16 |
2010-05-12_03:51:29.94145 kern.warn: ESI: f6ce1c00 EDI: df03781c EBP: df0377fc ESP: df0377a8 |
| 17 |
2010-05-12_03:51:29.94147 kern.warn: DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068 |
| 18 |
2010-05-12_03:51:29.94150 <0>Process fastcgi (pid: 32201, ti=df036000 task=c3c46e10 task.ti=df036000) |
| 19 |
2010-05-12_03:51:29.94152 <0>Stack: |
| 20 |
2010-05-12_03:51:29.94154 kern.warn: c04d35fb 00000000 00000000 00000000 00000000 df0377fc c04d39f7 00000000 |
| 21 |
2010-05-12_03:51:29.94156 <0> 000001f8 00000000 df037a54 df037940 00000004 00000001 000001f8 00000000 |
| 22 |
2010-05-12_03:51:29.94159 <0> 00000000 00000070 00000000 df037a80 effd8380 df0379d8 c04df769 00000070 |
| 23 |
|
| 24 |
The fastcgi process mentioned in report is perl script (but it uses C |
| 25 |
libraries libev and libadns for accessing epoll(2) and doing async dns |
| 26 |
resolving (it's event-based fastcgi implementation, not usual forked one). |
| 27 |
|
| 28 |
Server was rebooted, now everything is fine. Server software is nearly |
| 29 |
up-to-date x86 Gentoo (last update was 2-3 weeks ago), kernel is |
| 30 |
sys-kernel/hardened-sources-2.6.28-r9. |
| 31 |
|
| 32 |
-- |
| 33 |
WBR, Alex. |
Archives