Gentoo Archives: gentoo-user

From: Grant <emailgrant@×××××.com>
To: gentoo-user@l.g.o
Subject: Re: [gentoo-user] mysqld invoked oom-killer
Date: Thu, 21 Jul 2011 21:35:13
Message-Id: CAN0CFw1aF8Hjyc3Dh72Qf5iUaif-CJsUe_zhfe7iferq4GdOFw@mail.gmail.com
In Reply to: Re: [gentoo-user] mysqld invoked oom-killer by kashani
1 >>>> I ran into an out of memory problem.  The first mention of it in the
2 >>>> kernel log is "mysqld invoked oom-killer".  I haven't run into this
3 >>>> before.  I do have a swap partition but I don't activate it based on
4 >>>> something I read previously that I later found out was wrong so I
5 >>>> suppose I should activate it.  Is fstab the way to do that?  I have a
6 >>>> commented line in there for swap.
7 >>
8 >> ...
9 >>>
10 >>> If you're running any other servers that utilize MySQL like Apache or
11 >>> something, check its access logs to see if you had an abnormal number
12 >>> of connections. Bruteforce hacking or some kind of flooding/DOS attack
13 >>> might cause it to use more memory than it ordinarily would.
14 >>
15 >> I don't know why I didn't check the apache2 error log before, but I
16 >> got the following entry 2 seconds before the server became
17 >> unresponsive:
18 >>
19 >> [error] server reached MaxClients setting, consider raising the
20 >> MaxClients setting
21 >>
22 >> I use the default 256 for MaxClients.  This confirms the server was
23 >> brought down by too many child processes consuming too much memory.
24 >> Looking back at the access_log, it's clear this condition was caused
25 >> by the single IP which requested one of my pages about 300 times over
26 >> the course of 1 minute.  This caused my entire server to lock up for
27 >> hours until I rebooted it.
28 >>
29 >> I hesitate to reduce MaxClients from 256.  I think my server should be
30 >> able to handle it since it's the default.  So I need to prevent my
31 >> apache2 child processes from consuming so much memory?  apache2 was
32 >> restarted about an hour before the lockup so it had a pretty fresh
33 >> start.  I do use mod_perl which is a memory hog from what I
34 >> understand.  Do I just need more RAM?
35 >
36 >        Most people do not think about this correctly. Can your server run
37 > 10000 Apache processes? No, not enough resources. 1000? No, same problem.
38 > 256? I'd say no based on this thread. If you're not going to set it at 10000
39 > why try to keep it at 256?
40
41 This is illuminating, thank you. I wonder why the default is set at
42 256 though. Do my apache2 processes use an unusual amount of memory
43 at 25M-67M? Even at 25M each, you would need 6.4GB of memory for 256
44 processes. Is an apache2 restart the only thing that prevents apache2
45 processes from growing in memory usage indefinitely?
46
47 - Grant
48
49
50 >        Next image a grocery store with 256 checkout lanes, but only four
51 > cashiers. Four cashiers trying to run that many lanes is actually slower
52 > than having only four lanes. However 32 lanes could faster than 4. People
53 > can have their groceries setup, baggers aren't getting in the way, etc. The
54 > analogy breaks down a bit, but you get the point.
55 >
56 >        There is no performance gain in configuring for concurrency your
57 > hardware and software can not support.
58 >
59 > kashani