| 1 |
>>>>>>> My web server's response time for http requests skyrockets every |
| 2 |
>>>>>>> weekday between about 9am and 5pm. I've gone over my munin |
| 3 |
>>graphs |
| 4 |
>>>>and |
| 5 |
>>>>>>> the only one that really correlates well with the slowdown is |
| 6 |
>>"TCP |
| 7 |
>>>>>>> Queuing". It looks like I normally have about 400 packets per |
| 8 |
>>>>second |
| 9 |
>>>>>>> graphed as "direct copy from queue" in munin throughout the day, |
| 10 |
>>>>but 2 |
| 11 |
>>>>>>> to 3.5 times that many are periodically graphed during work |
| 12 |
>>hours. |
| 13 |
>>>>I |
| 14 |
>>>>>>> don't see the same pattern at all from the graph of all traffic |
| 15 |
>>on |
| 16 |
>>>>my |
| 17 |
>>>>>>> network interface which actually peaks over the weekend. TCP |
| 18 |
>>>>Queuing |
| 19 |
>>>>>>> doesn't rise above 400 packets per second all weekend. This is |
| 20 |
>>>>>>> consistent week after week. |
| 21 |
>>>>>>> |
| 22 |
>>>>>>> My two employees come into work during the hours in question, and |
| 23 |
>>>>they |
| 24 |
>>>>>>> certainly make frequent requests of the web server while at work, |
| 25 |
>>>>but |
| 26 |
>>>>>>> if their volume of requests were the cause of the problem then |
| 27 |
>>that |
| 28 |
>>>>>>> would be reflected in the graph of web server requests but it is |
| 29 |
>>>>not. |
| 30 |
>>>>>>> I do run a small MTU on the systems at work due to the config of |
| 31 |
>>>>the |
| 32 |
>>>>>>> modem/router we have there. |
| 33 |
>>>>>>> |
| 34 |
>>>>>>> Is this a recognizable problem to anyone? |
| 35 |
>>>>>> |
| 36 |
>>>>>> |
| 37 |
>>>>>> I'm in the midst of this. Are there certain attacks I should |
| 38 |
>>check |
| 39 |
>>>>for? |
| 40 |
>>>>> |
| 41 |
>>>>> |
| 42 |
>>>>> It looks like the TCP Queuing spike itself was due to imapproxy |
| 43 |
>>which |
| 44 |
>>>>> I've now disabled. I'll post more info as I gather it. |
| 45 |
>>>> |
| 46 |
>>>> |
| 47 |
>>>>imapproxy was clearly affecting the TCP Queuing graph in munin but I |
| 48 |
>>>>still ended up with a massive TCP Queuing spike today and |
| 49 |
>>>>corresponding http response time issues long after I disabled |
| 50 |
>>>>imapproxy. Graph attached. I'm puzzled. |
| 51 |
>>>> |
| 52 |
>>>>- Grant |
| 53 |
>>> |
| 54 |
>>> Things to check for: |
| 55 |
>>> Torrent or other distributed downloads. |
| 56 |
>>> Download program with multiple download threads |
| 57 |
>> |
| 58 |
>> |
| 59 |
>>There sure shouldn't be anything like that running either on the |
| 60 |
>>server or in the office. Is there a good way to find out? Maybe |
| 61 |
>>something that would clearly indicate it? |
| 62 |
>> |
| 63 |
>> |
| 64 |
>>> Maybe another proxy running? Esp. as you saw this also with |
| 65 |
>>imapproxy. |
| 66 |
>> |
| 67 |
>> |
| 68 |
>>nginx acts as a reverse proxy to apache2 but that's a pretty common |
| 69 |
>>config. Nothing else that I know of. |
| 70 |
>> |
| 71 |
>>- Grant |
| 72 |
> |
| 73 |
> Any way to find out between which hosts/servers those connections are for? |
| 74 |
> That might help in locating the cause. |
| 75 |
> |
| 76 |
> Eg. which of your desktops/laptops inside your network and where they are trying to connect to. |
| 77 |
|
| 78 |
|
| 79 |
The spikes are taking place on my remote server but they seem to |
| 80 |
roughly coincide with user activity within my own network. My |
| 81 |
technical knowledge of networking internals is weak. Does anyone know |
| 82 |
which tool will tell me more about the connections that are causing |
| 83 |
the TCP Queuing spikes? |
| 84 |
|
| 85 |
- Grant |
Archives