[gentoo-amd64] Re: Memory usage; 32 bit vs 64 bit.

[Duncan]

Alex Alexander posted on Mon, 03 Jan 2011 21:37:08 +0200 as excerpted:

> On 3 Jan 2011, at 20:34, Dale <rdalek1967@×××××.com> wrote:
>
>> I recently built me a new 64 bit system.  My old 32 bit system has 2Gbs
>> and my new system has 4Gbs.  I was expecting it to use about the same
>> amount of memory but noticed it uses a good bit more on the new system
>> than the old one.   With just the normal stuff open, I use about 1.5Gbs
>> of ram.  My old system would use a little over half that.   I have the
>> same settings on both.
>>
>> Is this difference because 64 bit programs use more memory, maybe they
>> are larger than 32 bit programs?  Just curious.  I notice that
>> Seamonkey uses more and KDE's plasma-desktop uses more.  Those are
>> generally the biggest users.
>>
>> I'm not complaining about the usage, just curious as to why the
>> difference.
>>
> Are you sure you're checking your free ram correctly? run "free" and
> check the buffers/cache line :)

Linux memory usage is notoriously confusing for the uninitiated and not 
entirely simple to explain or figure out the "real" per-app usage even for 
those who know /something/ about it.

First, to directly answer the question.  64-bit memory usage /will/ be 
somewhat higher, yes, but shouldn't be double.  The reason usage is higher 
is because address pointers are now 64-bit, not 32-bit, so /they/ take 
twice the space.  However, according to the gcc manpage:

       -m32
       -m64
           Generate code for a 32-bit or 64-bit environment.
           The 32-bit environment sets int, long and pointer
           to 32 bits and generates code that runs on any i386
           system.  The 64-bit environment sets int to 32 bits
           and long and pointer to 64 bits and generates code
           for AMD's x86-64 architecture. 

So the common "utility integer" standard C/C++ int types remain 32-bit.  
This actually one of the bigger issues in porting sources from 32-bit to 
64-bit, as for years, lazy 32-bit-only programmers were used to thinking 
of int, long and (memory) pointer as the same size, 32-bits, and being 
able to directly convert between them and use them nearly interchangeably, 
but that's no longer possible on amd64, because pointers and ints are no 
longer the same size.

But the point (not pointer! =:^) we're interested in for purposes of this 
discussion is that the very commonly used "utility integer" known simply 
as "int" remains 32-bit.  Because the 32-bit int is /so/ commonly used, to 
the point that it's the "default" integer type even on 64-bit, with only 
memory pointers and integers requiring 64-bit size getting full 64-bit, 
memory usage doesn't normally double, only increasing by some smaller 
factor, depending on the app and its particular mix of 32-bit int vs 64-
bit memory pointer and 64-bit long integers.

This additional memory usage is one of the negatives of 64-bit, and the 
reason that on archs other than x86, it's common to see 64-bit kernels for 
the ability to address > 4GB at the system level, with a 32-bit user-land 
since few individual apps (with noted exceptions) actually benefit from 
being able to address > 1-4 GB of RAM in a single app.  (Note the 1-4 GB 
range.   This is due to the common user-space/kernel-space split of the 4 
GB address space on 32-bit systems, meaning individual apps may be limited 
to only a gig of usable user-address-space, depending on whether the split 
is 1:3/2:2/3:1 or separate 4GB spaces for user and kernel.  Of course full 
64-bit doesn't have to worry about this.)

x86 is somewhat different in this regard, however, because traditional 32-
bit x86 is known as a "register starved" architecture -- the number of 
available full-CPU-speed registers on 32-bit x86 is comparatively limited, 
forcing code to depend on slower L1 cache (tho that's still way faster 
than L2/L3, which is way faster than main memory, which is way faster than 
typical spinning-disk main storage) where other archs could be using their 
relative abundance of CPU registers.  When it was designing amd64, AMD 
pretty much (I'm not sure if exactly) doubled the number of registers in 
their 64-bit hardware spec as compared to 32-bit (where they kept the same 
limited number of registers for compatibility reasons), with the result 
being that on amd64/x86_64 the speed-boost from access to these additional 
available registers often more than offsets the negative of the 
comparative double-size memory pointers.  The precise balance, whether the 
cost of dealing with double-size memory pointers or the benefit of access 
to all those additional registers wins, depends on the app in question, 
but in general the benefit of the extra registers on amd64/x86_64 as 
opposed to x86_32/ia32 is sufficient that it's far less common to see the 
64-bit kernel, 32-bit userland that is often seen on other archs.

That takes care of the direct answer.  Now to expand on what Alex referred 
to and what I mentioned in my intro as well, the topic of measuring Linux 
memory usage in general.

The uninitiated will often look at "free memory" (the value in the Mem: 
line of the "free" command, run at the command line) on Linux, and wonder 
why it's so small -- why Linux seems to use so much memory.  But, as Alex 
mentioned, that line is rather misleading, again, to the uninitiated.

Linux, like most OSs, considers "empty" memory "wasted" memory.  If the 
memory is available to use, therefore, Linux, as other OSs, will try to 
use it for something, normally for disk cache, mainly, with a bit used for 
other "buffering" as well.  When/if the system needs that memory for other 
stuff (apps), the cache and buffers can be dumped.

The confusion comes not in this, but rather, in the number actually 
exposed as "free" memory, which can be two very different values, either 
the actual "free" (unused=wasted) memory, or the "free for use if 
needed" (including memory used for cache and buffers) memory, depending on 
how the OS chooses to present it.  On Linux, the "free" memory as reported 
by the "free" command on the Mem: line is the first (unused=wasted), while 
that on the -/+ buffers/cache line is the second (free for use if needed).

Swap, of course, can be thrown in as another factor, since within context 
that can be seen as the reverse of disk cache -- app memory swapped out to 
disk as opposed to disk data cached in memory.  Thus free's Swap: line.
It's worth noting here the existence of the Linux kernel's swappiness 
parameter, exposed in the filesystem as /proc/sys/vm/swappiness .  This 
file contains a number 0-100 (attempting to set it > 100 results in an 
error), 60 being the default, indicating the desired balance between 
swapping apps out to retain disk cache and keeping apps in memory thus 
having less room for disk cache.  0 means always prefer keeping apps in 
memory, dumping cache when needed to do so, 100 means always prefer 
dumping apps to swap, retaining cache if at all possible.

As mentioned, the kernel swappiness default is 60, slightly preferring 
cache to apps.  A common recommendation found on the net, however, is to 
lower swappiness to something like 20, preferring with some strength 
retention of apps in memory to retention of cache.

Here, OTOH, I run swappiness=100, because swap is striped across four 
disks, while most of the filesystem is RAID-1 mirrored on the same four 
disks, so swap I/O should be faster than rereading formerly cached data 
back in off disk.  And, at least with my current 6 gigs RAM, with 
PORTAGE_TMPDIR on tmpfs (which is reported in free's cache value) and with 
parallel merging parameters carefully controlled so that even with 
swappiness=100 I only end up a few MB (perhaps a couple hundred) into 
swap, swappiness=100 works very well for me.  I don't notice the bit of 
swapping, and typically when I'm done, I might have 16 or 32 MB swapped 
out, that stays that way until I swapoff -a or reboot, indicating that I 
don't really use that bit of swapped apps much anyway or it'd be swapped 
back in when I did.

If you wish to experiment with swappiness, you can cat it to see the value 
as a normal user, but of course only save/echo a new value to it as root.

When you're done experimenting, if you want to make a permanent change, 
add a line ...

vm.swappiness = 100

... to your /etc/sysctl.conf file.  (Other /proc/sys/* settings can be 
similarly set this way, or of course with a simple echo-redirect line in 
/etc/conf.d/local or the like.  You can google for info on most or all of 
the other files under /proc/sys/, if interested.)

OK, back from the swappiness detour, to memory usage.

What sort of memory usage is reasonable?  Of course that depends on what 
you do with your computer.  =:^)  But, as you know, I'm a KDE user as 
well, and of course a gentoo/amd64 user.  Currently, I have an uptime of a 
week, which was when I last synced and updated both Gentoo and the kernel 
(thus the week uptime, since I rebooted into the new kernel then).  So 
I've not done a full update since I rebooted, tho I did emerge a few new 
packages (phonon-vlc and dependencies, including vlc, I was running phonon-
xine and still have it installed, but decided to try vlc and phonon-vlc) a 
couple days ago.  Of course I'm in KDE (4.5.4) ATM.  With that general 
system state and keeping in mind that I have 6 gigs RAM (the -m tells free 
to report in MB):

$free -m
             total       used       free     shared    buffers     cached
Mem:          5925       3334       2590          0        319       1571
-/+ buffers/cache:       1443       4481
Swap:        20479          0      20479

So ~ 2.5 gigs is entirely unused (empty, effectively wasted, ATM), with 
the ~ 3.25 gigs of used memory split between ~ 1.4 gigs used for apps and 
~ 1.8 gigs of cached and buffer memory, currently used to store data that 
can be dumped to make room for actual apps, if necessary.

Tho in my experience, even the 1.4 gigs of app usage isn't entirely 
required.  It has been awhile ago now, but at one point I was running 1 
gig of total RAM, with no swap.  At that time, app-memory usage seemed to 
run ~ half a gig.  When I upgraded RAM to 8 gigs (I since lost a stick 
that I've not replaced, thus the current 6 gigs), app memory usage 
increased as well, to closer to a gig (IIRC it was about 1.2 gig after a 
week's uptime, back then, to compare apples to apples as they say), 
without changing what I was running or the settings.  So given the memory 
to use, the apps I run apparently use it, up to perhaps a gig and a half.  
But if they're constrained to under a gig, they'll be content with less, 
perhaps half a gig.  I'm not sure of the mechanisms involved there except 
that apps do have access to the memory info as well, and perhaps some of 
them are more liberal with their own caching (in-memory web-page cache for 
browsers, etc) and the like, given memory room to work with.  But there's 
clearly a point at which they have their fill, as at a gig of RAM, apps 
were using half of it (half a gig), while when I upgraded to 8 gig, 8 
times the RAM, app-memory usage only just over doubled.  I suspect 4 gigs 
and 8 gigs would have about the same usage, but below 4 gigs, the apps 
start to be a bit more conservative with their own usage.

That covers overall system memory usage.  But what about individual apps?

Individual app memory usage on Linux is unfortunately a rather complex 
subject.  Top is a useful app for reporting on and controlling (nicing, 
killing, etc) other apps.  Top's manpage has a nice description of the 
various memory related stats and how they relate to each other, so I'll 
refer you to that for some detail I'm omitting here.  Meanwhile, on non-
swapping systems, resident memory (top's RES column) is about as accurate 
a first-order approximation of app memory usage as you'll get, but it's 
only reporting physical memory, so won't include anything swapped out.  
Also, the memory one could expect to free by terminating that app will be 
somewhat less than resident memory, due to libraries and data that may be 
shared between multiple apps.  Top has a SHR (shared) column to report 
potentially shared memory, but doesn't tell you how many other apps (maybe 
none) are actually sharing it.  Some memory reporting apps won't count 
shared memory as belonging to the app at all, others (like top, AFAIK) 
report the full memory shared as belonging to each app, while still others 
try to count how many apps are sharing what bits, and divide the shared 
memory by the number of apps sharing it.  Which way is "right" depends on 
what information you're actually looking for.  If you want the app totals 
to match actual total memory usage, apportioned share reporting is the way 
to go.  If you want to know what quitting the app will actually free, only 
count what's not shared by anything else.  If you want to know how much 
memory an app is actually using, regardless of other apps that may be 
sharing it too, count all the memory it's using, shared or not.

Then there's swapping.  Due to the way Linux works, the data available on 
swapped out memory is limited.  To get all the normal data would require 
swapping all that data back in, rather defeating the purpose of swap, so 
few if any memory usage reporting utils give you much detail about 
anything that's swapped out.  For people with memory enough to do so, a 
swapoff (or simply running without swap at all) force-disables swap, thus 
making full statistics available, but as mentioned above, to a point, many 
apps will use more memory if it's available, conserve if it's not, so 
running without swap on systems that routinely report non-zero swap usage 
doesn't necessarily give a true picture of an app's memory usage with swap 
enabled, either.

Conclusion:  While the output of the free command (and by extension, other 
references to free memory in Linux) may initially seem a bit unintuitive, 
it's straightforward enough, once one understands what's there.  
Unfortunately, the same can't be said about individual application memory 
usage, which remains somewhat difficult to nail down and even more so to 
properly describe, even after one understands the basics.  

FWIW, however, I don't claim to be a programmer or to understand all that 
much beyond the basics.  Should someone believe I'm in error with the 
above, or if they have anything to add or especially if they have a 
reasonably accurate simpler way to describe things, please post!  I love 
to learn, and definitely do NOT believe I've reach my limit in learning in 
this area!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman