| 1 |
Am 20.09.2016 um 21:52 schrieb Grant: |
| 2 |
>>>>>>>> My web server's response time for http requests skyrockets every |
| 3 |
>>>>>>>> weekday between about 9am and 5pm. I've gone over my munin |
| 4 |
>>> graphs |
| 5 |
>>>>> and |
| 6 |
>>>>>>>> the only one that really correlates well with the slowdown is |
| 7 |
>>> "TCP |
| 8 |
>>>>>>>> Queuing". It looks like I normally have about 400 packets per |
| 9 |
>>>>> second |
| 10 |
>>>>>>>> graphed as "direct copy from queue" in munin throughout the day, |
| 11 |
>>>>> but 2 |
| 12 |
>>>>>>>> to 3.5 times that many are periodically graphed during work |
| 13 |
>>> hours. |
| 14 |
>>>>> I |
| 15 |
>>>>>>>> don't see the same pattern at all from the graph of all traffic |
| 16 |
>>> on |
| 17 |
>>>>> my |
| 18 |
>>>>>>>> network interface which actually peaks over the weekend. TCP |
| 19 |
>>>>> Queuing |
| 20 |
>>>>>>>> doesn't rise above 400 packets per second all weekend. This is |
| 21 |
>>>>>>>> consistent week after week. |
| 22 |
>>>>>>>> |
| 23 |
>>>>>>>> My two employees come into work during the hours in question, and |
| 24 |
>>>>> they |
| 25 |
>>>>>>>> certainly make frequent requests of the web server while at work, |
| 26 |
>>>>> but |
| 27 |
>>>>>>>> if their volume of requests were the cause of the problem then |
| 28 |
>>> that |
| 29 |
>>>>>>>> would be reflected in the graph of web server requests but it is |
| 30 |
>>>>> not. |
| 31 |
>>>>>>>> I do run a small MTU on the systems at work due to the config of |
| 32 |
>>>>> the |
| 33 |
>>>>>>>> modem/router we have there. |
| 34 |
>>>>>>>> |
| 35 |
>>>>>>>> Is this a recognizable problem to anyone? |
| 36 |
>>>>>>> |
| 37 |
>>>>>>> I'm in the midst of this. Are there certain attacks I should |
| 38 |
>>> check |
| 39 |
>>>>> for? |
| 40 |
>>>>>> |
| 41 |
>>>>>> It looks like the TCP Queuing spike itself was due to imapproxy |
| 42 |
>>> which |
| 43 |
>>>>>> I've now disabled. I'll post more info as I gather it. |
| 44 |
>>>>> |
| 45 |
>>>>> imapproxy was clearly affecting the TCP Queuing graph in munin but I |
| 46 |
>>>>> still ended up with a massive TCP Queuing spike today and |
| 47 |
>>>>> corresponding http response time issues long after I disabled |
| 48 |
>>>>> imapproxy. Graph attached. I'm puzzled. |
| 49 |
>>>>> |
| 50 |
>>>>> - Grant |
| 51 |
>>>> Things to check for: |
| 52 |
>>>> Torrent or other distributed downloads. |
| 53 |
>>>> Download program with multiple download threads |
| 54 |
>>> |
| 55 |
>>> There sure shouldn't be anything like that running either on the |
| 56 |
>>> server or in the office. Is there a good way to find out? Maybe |
| 57 |
>>> something that would clearly indicate it? |
| 58 |
>>> |
| 59 |
>>> |
| 60 |
>>>> Maybe another proxy running? Esp. as you saw this also with |
| 61 |
>>> imapproxy. |
| 62 |
>>> |
| 63 |
>>> |
| 64 |
>>> nginx acts as a reverse proxy to apache2 but that's a pretty common |
| 65 |
>>> config. Nothing else that I know of. |
| 66 |
>>> |
| 67 |
>>> - Grant |
| 68 |
>> Any way to find out between which hosts/servers those connections are for? |
| 69 |
>> That might help in locating the cause. |
| 70 |
>> |
| 71 |
>> Eg. which of your desktops/laptops inside your network and where they are trying to connect to. |
| 72 |
> |
| 73 |
> The spikes are taking place on my remote server but they seem to |
| 74 |
> roughly coincide with user activity within my own network. My |
| 75 |
> technical knowledge of networking internals is weak. Does anyone know |
| 76 |
> which tool will tell me more about the connections that are causing |
| 77 |
> the TCP Queuing spikes? |
| 78 |
> |
| 79 |
> - Grant |
| 80 |
> |
| 81 |
> |
| 82 |
|
| 83 |
wireshark or whatever it is called at the moment? |
Archives