Re: [gentoo-user] Re: TCP Queuing problem

[Grant]

> It turned out this was a combination of two problems which made it
> much more difficult to figure out.
>
> First of all I didn't have enough apache2 processes.  That seems like
> it should have been obvious but it wasn't for two reasons.  Firstly,
> my apache2 processes are always idle or nearly idle, even when traffic
> levels are high.  But it must be the case that each request made to
> nginx which is then handed off to apache2 monopolizes an apache2
> process even though my backend application server is the one using all
> the CPU instead of apache2.  The other thing that made it difficult to
> track down was the way munin graphs apache2 processes.  On my graph,
> busy and free processes only appeared as tiny dots at the bottom
> because apache2's ServerLimit is drawn on the same graph which is many
> times greater than the number of busy and free processes.  It would be
> better to draw MaxClients instead of ServerLimit since I think
> MaxClients is more likely to be tuned.  It at least appears in the
> default config file on Gentoo.  Since busy and free apache2 processes
> were virtually invisible on the munin graph, I wasn't able to
> correlate their ebb and flow with my server's response times.
>
> Once I fixed the apache2 problem, I was sure I had it nailed.  That's
> when I emailed here a few days ago to say I think I got it.  But it
> turned out there was another problem and that was Odoo (formerly known
> as OpenERP) which is also running in a reverse proxy configuration
> behind nginx.  Whenever someone uses Odoo on my server, it absolutely
> destroys performance for my non-Odoo website.  That would have been
> really easy to test and I did test stopping the odoo service early on,
> but I ruled it out when the problem persisted after stopping Odoo
> which I now realize must have been because of the apache2 problem.


The root of the Odoo problem was that I didn't have keepalive enabled
between the nginx reverse proxy server and the Odoo server.  nginx
enables keepalive by default for the client side (HTTP/1.1) but not
for the upstream side (HTTP/1.0).  I still see TCP Queuing spikes in
munin with Odoo usage, but they no longer slow down the apache2/nginx
reverse proxy running my main site.

- Grant