[gentoo-amd64] Re: Advice On Enabling Transparent Hugepage

[Duncan]

Frank Peters posted on Thu, 02 May 2013 16:33:09 -0400 as excerpted:

> During kernel configuration there is an option called
> CONFIG_TRANSPARENT_HUGEPAGE which allows the use of Transparent
> Hugepages or application memory pages larger than the traditional 4K
> size.  This option has been available for a while but I've never enabled
> it.  However, it would seem like a good idea to use.
>
> Searching for more information I can find no comments on actual Hugepage
> performance for a simple desktop Linux system.  Can anyone verify or
> refute the idea that Transparent Hugepages will lead to improvements on
> a desktop system?  Does an application have to specifically request
> Hugepages or does the allocation occur for all applications?

In addition to what Paul says (which is generally correct, AFAIK)...

As I understand it, the whole idea of "huge pages" is based on making 
better use of a relatively limited hardware resource, the TLB entries, 
translation lookaside buffer entries.  The TLB is a sort of memory 
mapping cache index, the memory mapping itself being between virtual and 
physical memory.  The technical details of TLB are beyond me, so I'll 
simply refer you to wikipedia:

http://en.wikipedia.org/wiki/Translation_lookaside_buffer

But basically, the TLB is an index with a limit of (from wikipedia) 
12-4096 entries, depending on architecture and specific CPU.

The key is that each memory page requires an entry, and normal memory 
pages are 4k in size, so normal TLB coverage is relatively small, a few 
megabytes of RAM can be indexed at a time, that's it.

Which is where huge pages come in, since each huge page entry requires 
the same index line as a normal page would, but indexes and allows fast 
access to a far larger bit of memory, often 2 MB.  If all TLB line 
entries were for huge-pages, then, coverage would be several GB instead 
of several MB.

The problem was to implement that in the kernel.  Originally, there was a 
huge page interface that individual apps could use, but only if they 
specifically requested it, which of course required that they be 
rewritten in ordered to do so.

Which is where the transparent comes in.  With transparent huge-pages, 
apps work as normal -- it's the kernel that manages the huge pages 
"transparently".

Practical value depends on your work load, but in general, it won't hurt.

Two further notes: 1) Because I regularly run mainline git kernels, I was 
exposed to the option early on, and based on the coverage in (2) below, I 
enabled them.  But at one point in a pre-release kernel there was a bug, 
of course long since fixed.  The details aren't important as it is long 
fixed, but due to that bug, I *DID* discover ONE app that ends up using 
huge-pages a LOT -- firefox!  Evidently, huge pages only really come into 
play when individual app memory usage is relatively high, and firefox hit 
that threshold, as well as some other requirements for huge pages, plus 
some corner-cases that were required to hit this particular bug (which 
was with zero-pages, in the second kernel that had huge-page support as 
they added zero-page support, a different "trick" that wasn't initially 
applied to huge-pages, only normal pages).

So for that period, I simply disabled huge-page support again, until the 
bug was fixed.  Then I enabled it and haven't disabled it since.

2) LWN has several articles covering kernel huge-page support as well as 
transparent-huge-pages, including a 5-part-series covering the initial 
non-transparent version, the first part (with further links to the other 
parts) of which is found here:

http://lwn.net/Articles/374424/

That'll let you get pretty deep into the technology, if you like.  
There's benchmarks, ways to discover what the TLBs are for your hardware, 
etc.

Here's a google search limited to site:lwn.net for transparent huge pages:

http://www.google.com/search?q=site%3Alwn.net+transparent+huge+pages

Of interest, down the first page of hits a bit you can see overage of the 
"adding a huge zero page" discussion, the source of the bug I mentioned 
above, again, now long fixed.

That should give you quite enough information to get you started. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman