[gentoo-amd64] Re: Re: Hanging after a few days - 2.6.15-r7

[Duncan]

Brett Johnson posted <20060314135750.GC7383@××××.com>, excerpted below, 
on Tue, 14 Mar 2006 07:57:51 -0600:

> I'd like to take a minute to thank Duncan, as I usually learn something
> new with his very informative posts! Thank you for taking the time to
> write such detailed posts, I for one look forward to reading them.

Thanks.  Just remember that while I /try/ to make a distinction between
what's my opinion and what's fact as I know it, sometimes that fact as I
know it is wrong.  =8^(  That's happened a couple times recently. =8^(

I /do/ try to just as publicly acknowledge when I'm wrong, tho, so
hopefully if I've steered anybody wrong, they see my later nack too.

> On Mon, Mar 13, 2006 at 04:25:05PM -0700, Duncan wrote:
>> [With a kernel bug, first, see if it's in vanilla/mainline or not by
>> trying mainline, then using the instructions in the kernel BUG-HUNTING
>> doc to isolate it.
>>
> That is great information, and I will pursue  this if I do indeed have
> some memory anomaly.

Interestingly enough,  I just discovered BUG-HUNTING and used the
techniques therein to file my first kernel bug, isolated to a specific
line in 2.6.15-git11, that is, a change that happened between git10 and
git11, the 10th and 11th snapshots after 2.6.15 release, but before
2.6.16-rc1.  So... new knowledge fresh on my mind to share! =8^)  (The bug
is http://bugzilla.kernel.org/show_bug.cgi?id=6130 , and as reported in my
last comment to it, reversion of the culprit code in -rc6 has eliminated
the problem for now, altho they didn't fully trace the problem.  It's
rolled back to working for .16, after which they'll tackle the problem
agaiin for .17.)

>> You didn't mention this so I'm not sure if you know to make the
>> distinction or not -- what sort of memory usage was it?  Application
>> usage (indicates a leak) or simply cache or buffer usage?

> I also use the free command to view memory. I have been under the
> assumption that the "-/+ buffers/cache" line in the free command shows
> the actual memory "used" by the kernel and applications, and the "free"
> memory available for cache/buffers. This is how I have always based my
> assessment of memory usage.
>
> Here is a sample from this morning with just mutt, irssi, gaim and aterm
> running with fluxbox as the wm:
>
>             total    used    free  buffers  cached
> Mem:      1028780  996560   32220   234996   202852
> -/+ buffers/cache: 558712  470068
> Swap:      979956     224  979732
>
> I have thought that me "Mem: used" should always be as high as possible,
> meaning I am using as much ram in the system for applications as well as
> cache and buffers. I assumed the" -/+ buffers/cache used" is how much
> memory is consumed by the kernel and running applications. The "-/+ used
> number" is the one that I have seen grow overnight unusually high. This
> is also the number that conky seems to report on. Prior to "the change",
> and after a reboot, running the normal applications listed above, I
> average about 120000 on the "-/+ buffers/cache used". Maybe I am
> interpreting this data incorrectly, and I in fact do not have a problem.

No, you are interpreting it correctly, AFAIK.  I was concerned that you might
be using the top numbers and thinking that "free" was what you really had
available to use.  Common mistake, but one you obviously got past some time
ago, as you're interpreting the numbers correctly. =8^) but also =8^( 'cause
that means you have a problem!

> As a side bar, I saw mention in this list before, I think by Duncan, a
> "recommended" book on the Linux kernel. Unfortunately, it seems I
> deleted that thread, so if someone knows a good book on the kernel and
> memory management, I would appreciate any recommendations.

It wasn't me, but I remember seeing the mention as well.  Unfortunately, I
do /not/ remember what the recommendation was. =8^(

>> Kernel 2.6 has a swappiness tweaking control ( /proc/sys/vm/swappiness
>> ), that determines the balance between keeping stuff cached and
>> )swapping out more applications, once all memory is used.  This  is set by
>> default to 60.
>>
> Again, I learned something new today (and it's not even 8am here!), and
> I always kind of wondered why Duncan had some much swap and how he
> utilized it. This makes sense, and I will try tweaking this setting, as
> normally I don't use any swap.

I happened across that on the LWN weekly kernel page. =8^)  Andrew M (IIRC)
had the featured quote.  Apparently, there had been a lot of complaints
about the issue as I had explained above (apps left running overnight not
very responsive the first time they were used, and similar), and the featured
quote was Andrew saying he was going to "stick his fingers in his ears and
sing na-na-na" until the complainers could say they'd set swappiness to 0 and
were /still/ getting the problem!  <g>  As it happened, LWN either covered
the swappiness parameter that week, or had done so in one of the last couple
issues, so I learned what it was all about, but the reason it stuck as
effectively as it did was because of that quote! =8^)

As for my own swap, for awhile I was running with swap entirely disabled.
With a gig of memory for desktop use, it's possible, and the swap management
code /is/ pretty invasive and complicated, so if it's not necessary, turning
it off /should/ mean a slightly more efficient kernel, and will /certainly/
mean less code that can go wrong.  At the time, I had unstable memory (cheap
generic stuff, couldn't quite handle the rated pc3200 400MHz clock, and no BIOS
setting to declock it until a later BIOS upgrade, the stuff runs rock-stable at
pc3000 383MHz clock), and figured the less unnecessary code there was, the
better I'd run.

I thought I'd upgrade memory first, since I was having issues with it, but
when the BIOS upgrade and declocking to pc3000 speeds solved that issue,
another soon popped up -- my AC died, a VERY BAD thing to happen in the
middle of the summer here in Phoenix, with 110-120 F highs ( up to 48 C ).
I replaced it, then started having issues with my hard drive, as it had
overheated.  So... I had been thinking about going RAID, and took the
opportunity to do so, 4x300 gig Seagate SATAs.

As I hadn't even formatted about a hundred gig of my 250 gig Maxtor that
overheated and I was worried about going bad (I got it before I began
losing stuff, but it had developed some bad sectors), and with the mixed
RAID on the 4x300s giving me > 600 gig of usable storage, I had plenty of
space to work with.  So, while apportioning it, I decided to go ahead and
apportion some space for swap once again.  The general rule of thumb for swap
is 2X physical memory, and I was still planning on a memory upgrade, so
I took that into account as well.  If I fill all 8 slots with 2-gig sticks,
that will give me 16 gigs of memory.  However, I couldn't figure any way in
my wildest dreams that I'd need 48 gig of memory, and I figured I might
never upgrade to the full 16 gig of physical memory anyway, so I decided
16 gig of swap was a decent compromise.

As I said, I have that evenly allocated, a 4 gig swap partition on each
drive, specifically set to the same priority in fstab, which the kernel
then handles by striping them.  Thus, just as with striped raid (raid-0),
if there's sufficient data being transferred, it's at the higher speed
of four disks rather than the lower speed of one.  At that speed, swapping
is actually not too bad!  The only time it gets bad is if there's other i/o
going on at the same time, and even then, the kernel splits the tasks up
quite well so I still usually get better performance with other i/o than
I would with swap alone  (no other  i/o) on a single disk.

RAID, as with the dual CPU, was a very good choice.  I'm more than satisfied
with the performance of both, as both the dual CPU and the 4-way RAID have
exceeded my performance expectations.

I was actually tempted not to worry about upgrading my memory at all, after
that.  However, I'm still wanting to be an AT, and being able to mount a tmpfs
and have it all in physical memory, large enough to run a couple emerges at a
time, will clear up my current biggest machine performance bottleneck.
Additionally, the next logical upgrade is to switch to dual-cores, thus
giving me 4 cores since I'm already running dual (socket) Opterons, and that
upgrade simply doesn't make sense unless I'm running at least a gig for each,
or 4 gigs total.  Thus, 8 gigs should be quite decent, and serve me well
even after I upgrade to dual-cores, late this year or early next if I'm lucky.

As I said, I ordered that 8 gig last week.  Hopefully, it will be here by
Wed, altho the status still was listed as in-warehouse this AM, and I ordered
2-day shipping, so it's now more likely to be Thursday.

So.. that'll leave me with 8 gig of physical memory, and 16 gig of 4-way swap.

BTW, other than during that xorg composite memory leak I mentioned, I haven't
seen memory usage go much beyond a gig into swap, 2 gigs total.  The other
six gigs should therefore allow me to run portage's tempdir in RAM, and
/still/ give me more cached filesystem than I have now.  =8^)  I'm looking
forward to trying it out!  =8^)

It'll also give me a chance to see how that memory hole for 32-bit PCI at
the top of the 4-gig memory space works.  I've read about it, and I see the
options in the BIOS and know that's one of the things the kernel's IOMMU
setting deals with.  Now, I'll get to see how it all actually works, and see if
how I /think/ it works actually /does/ work that way.  =8^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-amd64@g.o mailing list