[gentoo-amd64] Re: Hanging after a few days - 2.6.15-r7

[Duncan]

Brett Johnson posted <20060313142338.GA7383@××××.com>, excerpted below, 
on Mon, 13 Mar 2006 08:23:39 -0600:

> I have noticed a similar problem that appears to be a kind of memory
> leak. I had upgraded both my desktop (amd64) and laptop (x86) from
> 2.6.14 to 2.6.15 a few weeks ago. After the upgrade, I started noticing
> that first thing in the morning or after long periods of the system being
> idle, the memory usage would climb to around 550MB. This seemed odd, as
> my normal memory usage is around 120MB (using conky as my system monitor).
>
> I tried to identify what process was consuming all that ram, but nothing
> looked out of the ordinary. As I started wokring on the machine, the
> memory usage would slowly go down over the course of a few hours, and
> level off around 200MB (which is still more than normal).

Some observations...

I haven't noticed the memory issue here.  I run mainline kernel.org
sources, using package.provided to tell portage I have gentoo-sources
2.6.999 so it won't bother me.  I've been running 2.6.15 and then 2.6.15.4
until 2.6.16-rc6 came out, as there was a bug with earlier rcs and the
daily git patches  beyond 2.6.15-git10 ( my kernel bug is
http://bugzilla.kernel.org/show_bug.cgi?id=6130 ).

Generally, the first step in tracing such bugs is to see if it still
occurs with the mainline/vanilla kernels.  If not, it's a patch in
whatever portage kernel you are using, so a Gentoo bug should be filed. 
If it occurs with mainline, try at least to narrow it down to the rc where
the problem started, and file a kernel.org bug.  Preferably, narrow it
down further to the daily git snapshot that  did it.  There are
instructions for doing that  and even going further, narrowing it down to
the file and even procedure or line changed between the two git snapshots,
if you have the time to do so, in the doc/BUG-HUNTING text-file in your
kernel sources dir.  Narrowing it down beyond the file level takes a lot
of time and patience, but narrowing it down to the file isn't too bad, and
narrowing it down to the GIT snapshot isn't hard at all, since tarballs of
those are made available at kernel.org just as if it were a normal kernel.

You didn't mention this so I'm not sure if you know to make the
distinction or not -- what sort of memory usage was it?  Application usage
(indicates a leak) or simply cache or buffer usage?  Linux will normally
fill most of memory with cache and buffers, as that's more efficient than
having it remain unused.

Kernel 2.6 has a swappiness tweaking control ( /proc/sys/vm/swappiness ),
that determines the balance between keeping stuff cached and swapping out
more applications, once all memory is used.  This  is set by default to
60.  If you want more applications kept in memory at the expense of
caching, set it lower, 0 means always prefer keeping applications in
memory even if they aren't used, which was the 2.4 series default. 
Conversely, higher than 60 will favor cache more than applications, so
apps will tend to be swapped out quicker while cache is maintained.  I
have a four-disk RAID setup here, with swap at the same priority over all
of them, so my swap is pretty fast and I keep swappiness set to 100,
favoring cache over application memory some of which won't be used very
often anyway.  (I'm running RAID-6 for much of my system, for 2X
redundancy.  Thus, on a four-disk array, it's two-way striped, while swap
is four-way striped, so reading apps back in from swap will be faster than
having to read data back in off the RAID-6 if it has been flushed out of
cache, and swappiness=100 is more efficient.  Normal non-RAID single disk
usage would mean swap and file rereading off of disk would be roughly the
same speed, so the best performance would be closer to that 60 default
swappiness.)

One of the most common memory/swap scenarios overnight, is that various
cron jobs run, logrotate, makewhatis, slocate's database update, etc,
filling the cache with data not likely to be used again, while pushing
applications into swap.  Applications left running constantly then take a
bit to load the first time in the morning, as they have to load back in
from swap, replacing some of the cached data.  This is normal, but if you
are just going off of the free memory figure, which includes cache and
buffers, it'll look like there was a memory leak overnight, because it's
all in cache in the morning.  Just as you mentioned, usage over the day
will likely increase free memory after that.  Of course, the other
noticeable effect is that constantly loaded apps take longer to become
usable first time in the morning, as they swap back in.

There are  a couple ways to deal with this.  First, here, I decided that I
didn't use slocate enough to be worth the trouble, particularly since my
schedule is messed up enough that there's no consistent time when I'm
/not/ at the computer, such that I can setup a cron job to run the
database update at that point.  I therefore don't have slocate on my
system at all.  I use grep and find, or the find functionality in mc or
KDE, to search for files, if I need to, and don't have slocate on my
system at all.

Second, set that swappiness to something fairly low, or to 0, thus
favoring apps over cache.  Cache will still use memory available, but it
won't force apps to swap to do so.

Third, if you aren't already aware of the distinction, learn the
difference between the different types of "used" memory, cache and buffers
vs. application memory, in particular.  (FWIW because it's hard to find a
definition for buffers, the difference between cache and buffers is that
buffers are basically dedicated cache -- usable by one application not the
entire system, while cache is systemwide. That's not absolutely accurate,
but it's a practical/working definition. The two are often considered
together and should be, as their effect on memory is similar and
cumulative.)

...

Finally, one thing I've noticed that /does/ leak memory here, under the
wrong circumstances -- the composite extension to xorg.  I run KDE, and
its kompmgr, part of kwin, is the application at fault.  Every time I
recompile xorg-server (modular-X) or the like (libXcomposite), I need to
recompile kwin as well.  That seems to kill the memory leak.  If they get
out of sync, X will consume more and more (application) memory, pushing
into swap virtually everything else.  A combination of setting ulimit for
memory and swap usage appropriately, and killing kompmgr when memory usage
gets too high (the usage is in X not kompmgr, but killing kompmgr frees
it), keeps it manageable on a 1 gig memory system (soon to be 8 gig,
memory ordered Friday, hopefully in by Wed!), if I forget to recompile
kwin after xorg-server/libXcomposite.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-amd64@g.o mailing list