Re: [gentoo-user] mysqld invoked oom-killer

[Grant]

>> I ran into an out of memory problem.  The first mention of it in the
>> kernel log is "mysqld invoked oom-killer".  I haven't run into this
>> before.  I do have a swap partition but I don't activate it based on
>> something I read previously that I later found out was wrong so I
>> suppose I should activate it.  Is fstab the way to do that?  I have a
>> commented line in there for swap.
>
> Yes, just uncomment it and should be automatic. (you can use "swapon"
> to enable it without rebooting)

Got it.

>> Can anyone tell how much swap this is:
>>
>> /dev/sda2           80325     1140614      530145   82  Linux swap / Solaris
>>
>> If it's something like 512MB, that may not have prevented me from
>> running out of memory since I have 4GB RAM.  Is there any way to find
>> out if there was a memory leak or other problem that should be
>> investigated?
>
> That's 512MB. You can also create a swap file to supplement the swap
> partition if you don't want to or aren't able to repartition.

So I'm sure I have the concept right, is adding a 1GB swap partition
functionally identical to adding 1GB RAM with regard to the potential
for out-of-memory conditions?

> I'd check the MySQL logs to see if it shows anything. Maybe check the
> settings with regard to memory upper limits (Google it, there's a lot
> of info about MySQL RAM management).

Nothing in the log and from what I read online, an error should be
logged if I reach mysql's memory limit.

> If you're running any other servers that utilize MySQL like Apache or
> something, check its access logs to see if you had an abnormal number
> of connections. Bruteforce hacking or some kind of flooding/DOS attack
> might cause it to use more memory than it ordinarily would.

It runs apache and I found some info there.

> A Basic "what's using up my memory?" technique is to log the output of
> "top" by using the -b command. Something like "top -b > toplog.txt".
> Then you can go back to the time when the OOM occurred and see what
> was using a lot of RAM at that time.

The kernel actually logged some top-like output and it looks like I
had a large number of apache2 processes running, likely 256 processes
which is the default MaxClients.  The specified total_vm for each
process was about 67000 which means 256 x 67MB = 17GB???

I looked over my apache2 log and I was hit severely by a single IP
right as the server went down.  However, that IP looks to be a
residential customer in the US and they engaged in normal browsing
behavior both before and after the disruption.  I think that IP may
have done the refresh-100-times thing out of frustration as the server
started to go down.

Does it sound like apache2 was using up all the memory?  If so, should
I look further for a catalyst or did this likely happen slowly?  What
can I do to prevent it from happening again?  Should I switch apache2
from prefork to threads?

- Grant