[gentoo-amd64] Re: 4Gb of RAM Memory- High Memory

[Duncan]

"Juliano Morais Barbosa" <barbosa@×××××××××××××××××.br> posted
01a201c679e1$c73faf80$c902a8c0@t201, excerpted below, on  Wed, 17 May 2006
15:43:09 -0300:

> I have gentoo  2.6.16-gentoo-r1-Flexsolutions #1 SMP Thu Mar 30 08:21:07 BRT
> 2006 x86_64 AMD Athlon(tm) 64 Processor 3500+ AuthenticAMD GNU/Linux with
> 4Gb of RAM but in top I see 3Gb, and the kernel don´t have option for High
> Memory Option.

The latter (lack of a kernel himem option) is because amd64 has a flat
memory model, up thru some huge multi-terabyte number, anyway.

What you are seeing isn't therefore a problem with the kernel, but due to
certain hardware limitations and the BIOS config options that work around
them (and one kernel option that's indirectly related).

The problem is this:  The old 32-bit PCI system was designed to work in
32-bits (obviously), and as such can only directly address 32-bits of
memory.  As it happens, that's 4 gigabytes.  For compatibility with those
legacy PCI devices, there's a "memory hole" in the last quarter or half a
gig below the 4-gig boundary, designed such that the address space in that
hole is reserved for use by those legacy PCI devices, rather than for use
as memory.  

If you remember the bad old days of MS-DOS and sub-megabyte memory, this
memory hole is comparable to the 640K-1MB MS-DOS High Memory Area, which
could only be used by device drivers and the like, and formed a hole
between conventional memory (640K down) and the XMS or EMS memory living
above a megabyte.  Even if you weren't running an OS that was restricted
to 640KB of conventional memory, you pretty much had to observe that hole,
because that's where all the various hardware devices expected to be, and
where the motherboards themselves were hard wired (MTRR and hardware cache
control) to expect hardware access to be as well.

Your BIOS should have two options that control how memory, once you get
enough to be blocked by the memory hole (IOW, 4 gig), is presented to
software.  It's unfortunately rather difficult to take a look at the BIOS
while I'm writing this and get the wording exactly right, but there should
be an option that toggles between ignoring the hole, so part of your
memory isn't accessible, and remapping the affected memory into the above
4 GB area, effectively bringing it out from behind the PCI hardware
address space overlay that would otherwise hide it.

The second BIOS option again deals with MTRRs (which control hardware
caching) and how they describe the memory hole space, either Contiguous,
making no distinction between this space and memory (marking it cachable
just as is normal memory), or Discrete, treating the area as distinct from
memory (with a specific mapping marking it uncachable).  There are a
limited number of MTRRs and more of them are used up mapping it
discrete/uncachable, but certain hardware will have issues if it's marked
contiguous/cachable.  I'd suggest experimenting with this one.

Once you get the first BIOS option set correctly, you should be see all
your memory.  However, there's then a new problem.  DMA transfers data
efficiently between hardware on the bus and memory, but if that hardware
can only see 32-bits (4 gigs) worth of memory, and you now have memory
mapped above that, you now have memory that can't be directly addressed by
DMA.

This is where the kernel comes in.  Authentic AMD AMD64 CPUs can take
advantage of a feature in their hardware GART (Graphics Address Remapping
Table) to use it as a hardware IOMMU (i/o memory management unit,
basically a hardware memory remapper that allows DMA to access >4 gig as
if it was below the barrier instead of above it) as well, not just for
graphics, but for PCI bus hardware in general. Unfortunately, Intel CPUs
implementing x86_64/amd64 don't have this hardware, so they must emulate
it in software.  When compiling for x86_64, however, there's a single
kernel option that enabled both, handling it with the emulation on Intel,
taking advantage of the hardware on true AMD64.  In menuconfig, it's under
"Processor Type and Features". The exact description of the option there
has changed recently to make it clearer what it does.  In 2.6.17-rcX (and
IIRC 2.6.16, but not .15 or earlier, which used the earlier wording), the
option is "K8 GART IOMMU support". You'll probably want this enabled. Note
that with 2.6.16 and later, you'll almost certainly HAVE to have it
enabled if you use SATA or SCSI, as their drivers will locate above the 4G
boundary if there's memory up there to locate into, and at least in some
instances, the hardware will only do 32-bit, so you'll kernel panic as it
tries to load on boot, if you don't have that option on.  (Kernel 2.6.15
and previous used software bounce-buffers for the SCSI drivers, and
therefore libATA based SATA which uses SCSI, regardless.  Of course, this
eliminated many of the benefits of DMA, one reason they switched in 2.6.16
to going thru the GART/IOMMU where available.  The lack of such hardware
on Intel means they still use software bounce buffers, only now controlled
by the IOMMU instead of the SCSI layer.  Of course, this is one of the
ways Intel's em64t solution is inferior to true amd64, but honestly, most
won't see it as most aren't using 4 gig or greater memory yet anyway.)

There's a couple more catches to it, however.  The first one is back in
BIOS again. I'm not sure if it affects all chipsets, but here at least,
the AGP Fastwrite option is entirely incompatible with a working
GART/IOMMU.  I looked long and hard but never did find mention of that
elsewhere, and only found out by accident when I happened to twiddle the
IOMMU thing that never worked for me on, at the same time I twiddled the
BIOS AGP Fastwrite option off, and found both the GART and the IOMMU now
worked!

The second catch has to do with slaveryware video drivers.  Naturally, if
you are running a slaveryware GART, you won't be using the one in the
kernel, and won't be able to make use of the kernel's GART/IOMMU option. 
You'll have to look elsewhere for help in that case, as I know nothing
about it, beyond having read a (possibly misinformed) comment by an NVidia
user that their GART implementation didn't do the IOMMU stuff.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@g.o mailing list