[gentoo-amd64] Re: Managing CPU usage when doing emerges

[Duncan]

Branko Badrljica <brankob@××××××××××.com> posted
4A12AF80.8040705@××××××××××.com, excerpted below, on  Tue, 19 May 2009
15:09:20 +0200:

> Which is more or less ext4 performance if files remain in RAM. When
> memory pressure increases, ext4 wil just start writing to the files,
> which can actually be faster than tmpfs <--> swap transfers...

You bring up a very interesting point.  I wonder if anyone has 
benchmarked both in-memory and to-disk operation of ext4 with extreme 
write-out times and data=writeback, as compared to tmpfs, hitting swap 
for disk.

Now let me be plain, I wouldn't trust valuable data I need to keep around 
to ext4 yet, unless I was certain of my backups.  (Of course if it's 
valuable data you want to keep around, you have and have tested your 
backups in any case, AND keep them updated to the point at which you'd be 
comfortable losing data, whether that be a few minutes/hours worth or six 
months worth, but there's a LOT of folks that simply do not have backups 
at all, and ext4 isn't yet something I'd recommend to them unless they 
don't care about losing the data at all.) Maybe by 2.6.31, if .30 ext4 is 
demonstrated stable enough not to be still taking big tweaks in .31, but 
that's not yet the case with .29 as .30 has some pretty big changes.

However, that's NOT what we're talking here AT ALL.  Here, we're talking 
specifically temporary data, intended to be wiped before reboot, such 
that we don't care if it ever hits disk at all.  For that, ext4 is 
definitely MORE than acceptable at this point.

Here's the deal, tho.  With tmpfs, since the devs know it's not intended 
to hit disk /at/ /all/ unless as swap, there's whole sections of normal 
filesystem code that simply don't apply at all.  That's not a /lot/ of 
overhead when we're talking writing to the slow disk anyway, but it's a 
bit more significant when we're talking memory-speed operations.  That, 
and the flexibility of tmpfs to use swap when necessary, has 
traditionally made it an extremely good choice for such temporary data.

If we stop right there, ext4 would still have higher overhead, even with 
delayed write, and would thus be a characteristically worse choice.  
HOWEVER, and a BIG HOWEVER it is, there's a a particular caveat with swap 
where ext4 gets an extremely good mark, that may well turn the tables.

The problem with swap in general, and thus anything that ends up using 
it, is that it has sort of been the ugly stepchild no one has bothered to 
do a whole lot with.  The attitude of the kernel devs has seemed to be 
that with memory so cheap these days, there should be no reason to be 
using swap under normal operating conditions anyway; that it's only an 
extreme worst-case solution to exercise right before the OOM-killer goes 
to work, and that as a result, any serious work optimizing it beyond sane 
minimums is "wasted".  This has of course been somewhat controversial, 
particularly for users of older systems with < a gig of memory capacity, 
total, and only a single disk, but the fact is, swap /has/ been a 
somewhat ignored subsystem.

Obviously, ext4 is anything /but/ ignored.  Thus, it might actually be 
worth the (at least in theory) somewhat higher memory overhead if the 
disk access is significantly more optimized than swap disk access is, and 
as explained above, that's a reasonably likely case.

Of course, as you stated earlier in passing but no one has actually 
fleshed out, this assumes anything BUT default ext4 writeout expiry 
parameters.  We're talking expiry times on the order of an hour, instead 
of the default few seconds, thus keeping data in memory for the entire 
emerge, so the whole working dir can be created, the sources unpacked to 
it, the binaries compiled and whatever else needs done to the sources, 
fake-installed, qmerged to the live filesystem, then the whole thing, 
scratch files, sources, working dir, all of it, deleted, before the ext4 
write-out expiry occurs, assuming no memory pressure to cause the 
writeouts earlier.

But it's a valid question to raise and one I've seen no discussion of, 
anywhere, certainly not in the LWN and HOnline coverage that's where I 
get most of my kernel information from.  Under that sort of explicit 
temporary use only tweaking of the ext4 tuning knobs, exactly how /does/ 
it compare to tmpfs, both under memory-only scenarios and under memory-
pressure-therefore-swapped-or-written-out scenarios.  Benchmarks are 
definitely called for, as it's a very legitimate and unsettled question, 
going on theory alone.

Of course, in a typical single-disk usage scenario, with both a swap and 
this dedicated ext4 temp partition we're talking about, it would very 
likely depend on the partition ordering, with whichever partition is 
closest to the outside of the disk getting the better marks as the 
physical disk I/O will be faster to it.  But, for the benchmark the two 
partitions could be swapped, and that compared against having the OTHER 
one on a separate device, say swap on a different device when testing the 
ext4 partition, just so it didn't interfere.

Another typical case-by-case factor would be what other data is read/
written and how close it is to the partition in question.  (For our case, 
the Gentoo tree and sources and system header, etc files, read, the 
package database, live filesystem destinations, and binpkg location if 
that feature is turned on, write, and ccache both r/w if it's on, are the 
system data that would be accessed during the build.)  That sort of thing 
would have to be kept in mind and controlled for as well, in addition to 
the usual effort at keeping unrelated process activity to a minimum 
during the benchmark to avoid screwing the outcome.

Sounds interesting to think and read about, if someone has a link.  But 
honestly, I don't personally care enough about it to do the benchmarks or 
even to spend time looking to see if there are any, myself.

So, does it sound like fun to anyone?  For all I know someone's already 
tested it.  I haven't been following ext4 development close enough to 
know for sure, if I didn't simply happen across it.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman