| 1 |
On Wed, Jan 12, 2011 at 4:47 PM, Kaddeh <kaddeh@×××××.com> wrote: |
| 2 |
> Jarry, |
| 3 |
> |
| 4 |
> Thanks for the monitoring advice, I am checking out monit right now. |
| 5 |
> |
| 6 |
> In terms of what is the root cause of the issue, I have narrowed it down to |
| 7 |
> either write caching of a SQL cache issue. |
| 8 |
> |
| 9 |
> First, addressing the SQL issue and why I think that that could be one of |
| 10 |
> the causes. The entire site, for the most part is all in one giant DB |
| 11 |
> (~9GB) a significant part of that is a 3gb table full of raw image data |
| 12 |
> (yes, I know that this is a REALLY bad idea to do, but I didn't design the |
| 13 |
> site, I just did a migration to off-site) that being said, there could be a |
| 14 |
> problem with that. |
| 15 |
> |
| 16 |
> The write caching hteroy just comes up because I can clear the cached memory |
| 17 |
> down to 14mb cached using 'sync && echo 3 > /proc/sys/vm/dump_cache' |
| 18 |
> |
| 19 |
> Cheers |
| 20 |
> |
| 21 |
> Kad |
| 22 |
> |
| 23 |
> On Wed, Jan 12, 2011 at 1:37 PM, Jarry <mr.jarry@×××××.com> wrote: |
| 24 |
>> |
| 25 |
>> On 12. 1. 2011 19:59, Kaddeh wrote: |
| 26 |
>> |
| 27 |
>>> P4 @ 3.0Ghz |
| 28 |
>>> 2GB PC2 4200 |
| 29 |
>>> 2x 250GB drives in RAID1 |
| 30 |
>>> The system configurations are default for the most part with the server |
| 31 |
>>> running MySQL and Apache. |
| 32 |
>>> The problem that I am running into at this point, however is that the |
| 33 |
>>> machine seems to run out of memory and will segfault either apache or |
| 34 |
>>> mysql when does so, when apache segfaults, it is a recoverable error, |
| 35 |
>>> when mysql does it, mysql can't recover short of restarting it. |
| 36 |
>>> At this point, I have found a soft fix by running a cron job every 6 |
| 37 |
>>> hours or so to clear the cached memory, which seems to be the problem, |
| 38 |
>>> however, I would like to find a more permanent fix to this issue. |
| 39 |
>> |
| 40 |
>> First of all, find what is causing that excessive memory usage. |
| 41 |
>> I think 2GB should be enough for moderate web with apache+mysql. |
| 42 |
>> |
| 43 |
>> Second, use some monitoring software. Personally I'm using |
| 44 |
>> "monit" and I am very satisfied with it. It can monitor processes |
| 45 |
>> (if it is running, answering requests, etc), resources (disk, |
| 46 |
>> memory, swap, cpu, i/o), files (content, permissions, checksums), |
| 47 |
>> remote hosts (with some basic protocol checks i.e. http, ssh, |
| 48 |
>> smtp, ftp, mysql, ntp, dns...), it can inform you about problems |
| 49 |
>> (mail, log) and you can define rules what to do in case of anomalies |
| 50 |
>> (i.e. if mysql is using to much memory, it will be restarted). |
| 51 |
>> |
| 52 |
>> It can start/restart processes if they die (happened to me once |
| 53 |
>> with sshd on server which was ~50 miles away from me). You can |
| 54 |
>> put monit in inittab, so in case monit itself dies it is restarted |
| 55 |
>> automatically. Etc, etc. |
| 56 |
>> |
| 57 |
>> Jarry |
| 58 |
>> |
| 59 |
>> -- |
| 60 |
>> _______________________________________________________________ |
| 61 |
>> This mailbox accepts e-mails only from selected mailing-lists! |
| 62 |
>> Everything else is considered to be spam and therefore deleted. |
| 63 |
>> |
| 64 |
> |
| 65 |
> |
| 66 |
|
| 67 |
So, a few questions: |
| 68 |
|
| 69 |
What apache MPM are you using? You can control the number of processes |
| 70 |
or threads in that file. The default is something like 200 processes |
| 71 |
or threads (depending on MPM), so that could cause issues. |
| 72 |
|
| 73 |
What does your my.cnf look like? MySQL makes it pretty easy to |
| 74 |
regulate memory usage in my.cnf. |
| 75 |
|
| 76 |
What sort of webapp is this, PHP, python, perl, ...? |
| 77 |
|
| 78 |
That should be a good start. |
| 79 |
Cheers |
| 80 |
-- |
| 81 |
Matthew W. Summers |
Archives