Re: [gentoo-user] mysqld invoked oom-killer

[Michael Mol]

On Wed, Jul 20, 2011 at 7:54 PM, kashani <kashani-list@××××××××.net> wrote:
> On 7/20/2011 4:08 PM, Grant wrote:
>>>>
>>>> I ran into an out of memory problem.  The first mention of it in the
>>>> kernel log is "mysqld invoked oom-killer".  I haven't run into this
>>>> before.  I do have a swap partition but I don't activate it based on
>>>> something I read previously that I later found out was wrong so I
>>>> suppose I should activate it.  Is fstab the way to do that?  I have a
>>>> commented line in there for swap.
>>>
>>> Yes, just uncomment it and should be automatic. (you can use "swapon"
>>> to enable it without rebooting)
>>
>> Got it.
>>
>>>> Can anyone tell how much swap this is:
>>>>
>>>> /dev/sda2           80325     1140614      530145   82  Linux swap /
>>>> Solaris
>>>>
>>>> If it's something like 512MB, that may not have prevented me from
>>>> running out of memory since I have 4GB RAM.  Is there any way to find
>>>> out if there was a memory leak or other problem that should be
>>>> investigated?
>>>
>>> That's 512MB. You can also create a swap file to supplement the swap
>>> partition if you don't want to or aren't able to repartition.
>>
>> So I'm sure I have the concept right, is adding a 1GB swap partition
>> functionally identical to adding 1GB RAM with regard to the potential
>> for out-of-memory conditions?
>>
>>> I'd check the MySQL logs to see if it shows anything. Maybe check the
>>> settings with regard to memory upper limits (Google it, there's a lot
>>> of info about MySQL RAM management).
>>
>> Nothing in the log and from what I read online, an error should be
>> logged if I reach mysql's memory limit.
>>
>>> If you're running any other servers that utilize MySQL like Apache or
>>> something, check its access logs to see if you had an abnormal number
>>> of connections. Bruteforce hacking or some kind of flooding/DOS attack
>>> might cause it to use more memory than it ordinarily would.
>>
>> It runs apache and I found some info there.
>>
>>> A Basic "what's using up my memory?" technique is to log the output of
>>> "top" by using the -b command. Something like "top -b>  toplog.txt".
>>> Then you can go back to the time when the OOM occurred and see what
>>> was using a lot of RAM at that time.
>>
>> The kernel actually logged some top-like output and it looks like I
>> had a large number of apache2 processes running, likely 256 processes
>> which is the default MaxClients.  The specified total_vm for each
>> process was about 67000 which means 256 x 67MB = 17GB???
>>
>> I looked over my apache2 log and I was hit severely by a single IP
>> right as the server went down.  However, that IP looks to be a
>> residential customer in the US and they engaged in normal browsing
>> behavior both before and after the disruption.  I think that IP may
>> have done the refresh-100-times thing out of frustration as the server
>> started to go down.
>>
>> Does it sound like apache2 was using up all the memory?  If so, should
>> I look further for a catalyst or did this likely happen slowly?  What
>> can I do to prevent it from happening again?  Should I switch apache2
>> from prefork to threads?
>
>        Switching from prefork to threads and vice versa can be very
> difficult depending on which modules and libraries your site uses. It is not
> on the list of things you should try first. Or second. Maybe 37th.
>        I wouldn't expect adding swap to do much in this case. Your site gets
> hit hard, Mysql is a bit slow, Apache processes start stacking up, the
> system starts swapping, disk is really slow compared to RAM, and everything
> grinds to a complete halt possibly locking the machine up.
>
>        The easiest thing to try is to turn off keepalives so child processes
> aren't hanging around keeping connections up. Also lower the number of
> Apache children to 8 * number of processors or a minimum of 32. Test a bit.
> Turning off keep alive can cause problems for Flash based uploaders to your
> site and code that expect the connection to stay up. For most sites this
> shouldn't matter.
>
>        Next I'd look at tuning your Mysql config. If you've never touched
> my.cnf, by default it's set to use 64MB IIRC. You may need to raise this to
> get better performance. key_buffer and innodb_buffer_pool_size are the only
> two I'd modify without knowing more.

Also, run a caching proxy if at all possible. That made the single
biggest difference for my server.

Other useful things:
* Set the MaxRequestsPerChild to something like 450. As part of their
caching, things like mod_php will grow the process size a bit as the
apache process gets old in the tooth. Setting MaxRequestsPerChild
lower causes the process to expire and be replaced sooner. On my
server, I see apache processes consume about 60MB towards the end, and
then cycle back and consume about 22MB.
* On my server, I have MinSpareServers at 10, and MaxSpareServers at
12. I handle spikes pretty well, and free the memory quickly.
* If you're using PHP, set memory_limit in php.ini as low as your
applications can survive.

I'm assuming you're running on a VPS or similar. At 512MB of RAM with
a web server and database server, you need to keep things very tight.


-- 
:wq