Re: [gentoo-user] mysqld invoked oom-killer

[kashani]

On 7/20/2011 4:08 PM, Grant wrote:
>>> I ran into an out of memory problem.  The first mention of it in the
>>> kernel log is "mysqld invoked oom-killer".  I haven't run into this
>>> before.  I do have a swap partition but I don't activate it based on
>>> something I read previously that I later found out was wrong so I
>>> suppose I should activate it.  Is fstab the way to do that?  I have a
>>> commented line in there for swap.
>>
>> Yes, just uncomment it and should be automatic. (you can use "swapon"
>> to enable it without rebooting)
>
> Got it.
>
>>> Can anyone tell how much swap this is:
>>>
>>> /dev/sda2           80325     1140614      530145   82  Linux swap / Solaris
>>>
>>> If it's something like 512MB, that may not have prevented me from
>>> running out of memory since I have 4GB RAM.  Is there any way to find
>>> out if there was a memory leak or other problem that should be
>>> investigated?
>>
>> That's 512MB. You can also create a swap file to supplement the swap
>> partition if you don't want to or aren't able to repartition.
>
> So I'm sure I have the concept right, is adding a 1GB swap partition
> functionally identical to adding 1GB RAM with regard to the potential
> for out-of-memory conditions?
>
>> I'd check the MySQL logs to see if it shows anything. Maybe check the
>> settings with regard to memory upper limits (Google it, there's a lot
>> of info about MySQL RAM management).
>
> Nothing in the log and from what I read online, an error should be
> logged if I reach mysql's memory limit.
>
>> If you're running any other servers that utilize MySQL like Apache or
>> something, check its access logs to see if you had an abnormal number
>> of connections. Bruteforce hacking or some kind of flooding/DOS attack
>> might cause it to use more memory than it ordinarily would.
>
> It runs apache and I found some info there.
>
>> A Basic "what's using up my memory?" technique is to log the output of
>> "top" by using the -b command. Something like "top -b>  toplog.txt".
>> Then you can go back to the time when the OOM occurred and see what
>> was using a lot of RAM at that time.
>
> The kernel actually logged some top-like output and it looks like I
> had a large number of apache2 processes running, likely 256 processes
> which is the default MaxClients.  The specified total_vm for each
> process was about 67000 which means 256 x 67MB = 17GB???
>
> I looked over my apache2 log and I was hit severely by a single IP
> right as the server went down.  However, that IP looks to be a
> residential customer in the US and they engaged in normal browsing
> behavior both before and after the disruption.  I think that IP may
> have done the refresh-100-times thing out of frustration as the server
> started to go down.
>
> Does it sound like apache2 was using up all the memory?  If so, should
> I look further for a catalyst or did this likely happen slowly?  What
> can I do to prevent it from happening again?  Should I switch apache2
> from prefork to threads?

	Switching from prefork to threads and vice versa can be very difficult 
depending on which modules and libraries your site uses. It is not on 
the list of things you should try first. Or second. Maybe 37th.
	I wouldn't expect adding swap to do much in this case. Your site gets 
hit hard, Mysql is a bit slow, Apache processes start stacking up, the 
system starts swapping, disk is really slow compared to RAM, and 
everything grinds to a complete halt possibly locking the machine up.

	The easiest thing to try is to turn off keepalives so child processes 
aren't hanging around keeping connections up. Also lower the number of 
Apache children to 8 * number of processors or a minimum of 32. Test a 
bit. Turning off keep alive can cause problems for Flash based uploaders 
to your site and code that expect the connection to stay up. For most 
sites this shouldn't matter.

	Next I'd look at tuning your Mysql config. If you've never touched 
my.cnf, by default it's set to use 64MB IIRC. You may need to raise this 
to get better performance. key_buffer and innodb_buffer_pool_size are 
the only two I'd modify without knowing more.

kashani