Re: [gentoo-amd64] Re: Identifying CPUs in the kernel

[Peter Humphrey]

On Friday 22 June 2007 16:02:59 Duncan wrote:

> Was it you who posted about this before, or someone else?  If it wasn't
> you, take a look back thru the list a couple months, as it did come up
> previously.  You may have someone to compare notes with. =8^)

Nope, t'was I, and I've now tried lots more things but with no success.

> Separate processes or separate threads?  Two CPUs (um, two separate
> sockets) or two cores on the same CPU/socket?

Twin sockets, single Opteron 246 cores. And the BOINC scheduler sets off 
separate processes; they have distinct PIDs and on the old motherboard ran 
on distinct CPUs.

> There are [...] differences in architecture between AMD (with its onboard
> memory controller and closer cooperation, both between cores and between
> CPUs on separate sockets, due to the direct Hypertransport links) and
> Intel (with its off-chip controller and looser inter-core, inter- chip,
> and inter-socket cooperation).  There's also differences in the way you
> can configure both the memory (thru the BIOS) and the kernel, for separate
> NUMA access or unified view memory.  If these settings don't match your
> actual physical layout, efficiency will be less than peak, either because
> there won't be enough resistance to a relatively high cost switching
> between CPUs/cores and memory, so they'll switch off frequently
> with little reason, incurring expensive delays each time, or because
> there's too much resistance and to much favor placed on what the kernel
> thinks is local vs remote memory, when it's all the same, and there is in
> fact very little cost to switching cores/CPUs.

This board has eight DIMM sockets, four arranged next to each CPU socket and 
associated with it electrically and logically. I have four 1GB DIMMs, each 
in (I hope) the right pair of sockets in each bank of four. In other words, 
each CPU has 2GB of local RAM. I suppose I could buy as much RAM again and 
fill up all the sockets :-)

> If it's AMD with its onboard memory controllers, two sockets means two
> controllers, and you'll also want to consider NUMA, tho you can disable it
> and interleave your memory if you wish, for a unified memory view and
> higher bandwidth, but at the tradeoff of higher latency and less efficient
> memory access when separate tasks (each running on a CPU) both want to use
> memory at the same time.

NUMA  is switched on in BIOS and kernel config. I still find some of the 
BIOS settings mysterious though, so perhaps I don't have it set up right. I 
have tried the two sets of defaults, failsafe and optimised, but with no 
effect on this problem.

> In particular, you'll want to pay attention to the following kernel config
> settings under Processor type and features:
>
> 1) Symmetric multiprocessing support (CONFIG_SMP).  You probably have
> this set right or you'd not be using multiple CPUs/cores.

Yep.

> 2) Under SMP, /possibly/ SMT (CONFIG_SCHED_SMT), tho for Intel only, and
> on the older Hyperthreading Netburst arch models.

Nope.

> 3) Still under SMP, Multi-core scheduler support (CONFIG_SCHED_MC), if
> you have true dual cores.  Again, note that the first "dual core" Intel
> units were simply two separate CPUs in the same package, so you probably
> do NOT want this for them.

Nope.

> 4) Non Uniform Memory Access (NUMA) Support (CONFIG_NUMA) [...] You
> probably DO want this on AMD multi-socket Opteron systems, BUT note that
> there may be BIOS settings for this as well.  It won't work so efficiently
> if the BIOS setting doesn't agree with the kernel setting.

Yep.

> 5) If you have NUMA support enabled, you'll also want either Old style
> AMD Opteron NUMA detection (CONFIG_K8_NUMA) or (preferred) ACPI NUMA
> detection (CONFIG_X86_64_ACPI_NUMA).

Tried both together and each separately. No difference.

> 6) Make sure you do *NOT* have NUMA emulation (CONFIG_NUMA_EMU) enabled.

No point in emulating something that's present.  :-)

> What I'm wondering, of course, is whether you have NUMA turned on when
> you shouldn't, or don't have core scheduling turned on when you should,
> thus artificially increasing the resistance to switching cores/cpus and
> causing the stickiness.

I don't think so.

> If [...] you were correct when you said BOINC starts two
> separate /processes/ (not threads),

I'm sure I was correct.

> or if BOINC happens to use the older/heavier Linux threads model (which
> again will cause the threads to show up as separate processes),

I can't be quite certain this isn't happening, but I'm nearly so.

> There are two scheduling utility packages that include utilities to tie
> processes to one or more specific processors.
>
> sys-process/schedutils is what I have installed.  It's a collection of
> separate utilities, including taskset, by which I can tell the kernel
> which CPUs I want specific processes to run on.

> If you prefer a single do-it-all scheduler-tool, perhaps easier to learn
> if you plan to fiddle with more than simply which CPU a process runs on,
> and want to learn it all at once, sys-process/schedtool may be more your
> style.

I'll look into those - thanks.

-- 
Rgds
Peter Humphrey
Linux Counter 5290, Aug 93
-- 
gentoo-amd64@g.o mailing list