[gentoo-amd64] Re: upgrade an old system

[Duncan]

Sami Näätänen <sn.ml@××××××××××××.fi> posted
200905061637.59424.sn.ml@××××××××××××.fi, excerpted below, on  Wed, 06 May
2009 16:37:59 +0300:

> On Wednesday 06 May 2009 15:43:22 Mark Haney wrote:
>> Volker Armin Hemmann wrote:
>> > On Mittwoch 06 Mai 2009, Mark Haney wrote:
>> >> Duncan wrote:
>> >>> Consider pointing PORTAGE_TMPDIR at a tmpfs.
>> >>>
>> >>> The idea is based on the fact that everything portage does in its
>> >>> tmpdir (/var/tmp by default) is temporary, erased as soon as it's
>> >>> done emerging that package.  Since tmpfs uses swap backed memory,
>> >>> worst-case, it has to write to swap -- that is, to disk, which is
>> >>> where it would otherwise be writing ALL the temporary files.  With
>> >>> memory access so much faster than disk access, every file that's
>> >>> erased before it hits disk saves time, and it can make a BIG
>> >>> difference in emerge times.
>> >>
>> >> Duncan, you talk about tmpfs and I'm suddenly interested in trying
>> >> this out.  My question is, how much space do you allocate for the
>> >> tmpfs?  I know it'll fall back to swap if I'm out of space there,
>> >> but what works well for you?  I have (currently) 1GB RAM on this
>> >> system, but I'm getting ready to order more to get me to 4GB.
>> >
>> > 2gb. That is enough for almost everything. Not enough for openoffice.
>> > tmpfs                 2,0G  3,2M  2,0G   1% /var/tmp/portage tmpfs
>> >              1,0G  116K  1,0G   1% /tmp
>>
>> Now how is that going to play out when I only have 1GB of RAM?
>
> IMHO You shouldn't use tmpfs for the PM temp dir, until you can give
> that around 2GB, which shouldn't be more than half of your total memory.
>
> In your case there still could be some speed improvement if you have
> enough swap space. This because smaller packages wouldn't use disks at
> all, but still allowing bigger compiles to be carried out as the tmpfs
> would use swap when the memory starts to run low.
>
>
> When I had only 2GB of memory I used a dedicated RAID 0 (striped) ext2fs
> partition for the PM temp dir. As in this use case the raw speed of the
> used FS is more important than the reliability.
>
> Now I have 8GB of memory and I use 5GB and 2GB limits for the PM temp
> and the sys temp.

Sami posted pretty much what I would have posted.

As I said in my earlier post, at a gig of RAM, I'm honestly not sure if a 
tmpfs based PORTAGE_TMPDIR is worth it or not, as I've not personally 
tested it and have read no reports from others who have.  You will very 
likely still get faster compiles as some of the files won't be written to 
disk at all, but the tradeoff in terms of what gets swapped out and 
resulting system (lack of) responsiveness may or may not be worth it.  
There's some chance you could improve responsiveness by tweaking 
swappiness, but at a gig of memory, I expect it's still going to be very 
much a personal preference as to whether you consider the sacrificed 
system responsiveness worth the faster compiles, or not.  I suspect those 
that setup their emerges for overnight or otherwise set them up and then 
do something other than work on that computer will find the faster 
compiles useful, while those that try to do something else on the 
computer while it's compiling are more likely to find the decreased 
system responsiveness not worth it.

As for tmpfs size, there's a couple basics to keep in mind.

First, unlike some ramfs implementations (I'm not sure about the 
kernel's) tmpfs doesn't actually use memory unless there's files actually 
stored in it.  Thus, the size of the tmpfs is really its maximum size -- 
the size you as sysadmin are prepared to let it grow to.  Restating 
directly just to make it clear, an empty tmpfs uses very little memory at 
all (possibly a couple kB for the base memory structures), so unless it's 
actually used, the (max) size of the tmpfs doesn't matter.

Second, it's worth keeping in mind the kernel tmpfs default (max) side as 
a guideline, if nothing else.  It defaults to half of physical memory.  
Thus, create a tmpfs without setting its size, and if you have a gig of 
RAM, it'll default to a half gig; if you have 8 gigs, it'll default to 4 
gigs.  Again, for practical purposes that's max size, as if it remains 
unused, it takes virtually no memory at all.  (See the mount (1) manpage 
under tmpfs for this and other details.  Wow!  Somewhat OT, but I just 
checked the mount manpage to double-check my half-of-memory assertion, 
and discovered the new mpol= NUMA node allocation policy.  Interesting!)

Third, tmpfs is useful in that it isn't restricted to physical memory, 
and can use swap as well, if there's memory pressure and something in 
tmpfs to swap out.  Thus the "worst case" as mentioned earlier, that 
there's not enough memory and the system has to write the files to disk 
(swap, now) after all.  But another implication, then, is that swap size 
matters too.  It can't use swap you don't have configured and mounted!  
The old rule of thumb was to have swap of twice the size of regular 
memory.  While that no longer really applies as it used to, because if 
someone has memory of say four gigs, it's unlikely they're going to be 
prepared to wait for 8 gigs of swap to fill up even if they have it 
configured (at least if they're running a single disk, RAID-0/striped 
swap is faster and thus not as bad), for those with a gig of RAM or more, 
I'd say 2 gigs swap minimum is reasonable.  Those using tmpfs for 
anything major, however, as we're talking about here, will probably want 
more, say 4-6 gigs swap, just in case.

(As it happens, I have 8 gigs RAM and 16 gigs swap, consisting of 4,
4-gig swap partitions, one each on four separate drives, configured as 
RAID-0/striped, for speed.  16 gigs swap is a bit excessive, but as I use 
one of those 4-gig stripes as a suspend image, with a suspend image up to 
half RAM size or four gigs, and the kernel doesn't like striped suspend 
images very well, it works out quite well.)

With those basics in mind, we can now discuss tmpfs size policy.  For PM 
temp dir, tmpfs or not, that filesystem needs to be big enough to handle 
your biggest package, possibly more if you compile multiple packages in 
parallel.  The biggest actually compiled package is, I believe, open-
office.  Depending on options, I've seen it said (I don't have it 
installed, here) that open-office takes 4-5 gigabytes of filesystem space 
to succcessfully compile.  Thus, that's what I used as my baseline here, 
even tho I don't expect to install it.  (FWIW, I do have parts of koffice 
installed, tho.)  It's also worth noting that I saw one of the games 
developers remark that there's one games data package that's some 25 gigs 
(!!) in size.  However, it's always possible to point PORTAGE_TMPDIR (or 
other PM tmpdir, if you're using pkgcore or paludis) at some other 
location for huge packages, if necessary.  Thus, the tmpfs doesn't 
actually HAVE to be a full 5 gigs.  If it's too small, you'll get an 
error during the build and can either increase it temporarily (if you 
have the memory and swap available to do so), or can point the PM tmpdir 
elsewhere for that package from that point on.  Of course, it gets old 
VERY fast doing that for more than a handful of packages, so you'll want 
a tmpfs of at LEAST 2-3 gigs, if not the 5 gigs I used as a baseline for 
mine, to point PORTAGE_TMPDIR at.

But this is another reason it gets iffy running this way with only a gig 
of RAM.  If you're compiling a package needing 3 gigs filespace while 
compiling, with only a gig of RAM and assuming a half-gig RAM of system 
usage, you'll be into swap by two and a half gigs just for the tmpfs 
usage while compiling that package!  Yes, it's still likely to be a bit 
faster than building to a disk based filesystem, but probably not so 
much, and meanwhile, running that far into swap is likely to make system 
responsiveness rather bad.  Is it worth the build-time speedup for the 
loss of responsiveness?  How bad will it actually be?  Can tweaking 
swappiness help enough to make it usable?  These are questions I don't 
have the answer to, as I've yet to see a good report from anyone who 
actually tried it.

OK, so we've established a 5 GB reasonable tmpfs size for the PM tmpdir.  
FWIW, here, with as mentioned 8 gig RAM and 16 gig swap (enough I 
basically never have to worry about running out of swap), and with that 5 
GB open-office baseline, I decided a 6 GB (max size) tmpfs was 
reasonable.  Worst-case, full-maxed-out, that'd leave the system a 
minimum of 2 gigs of RAM to play with, before it has to decide whether to 
start swapping stuff out.  That seemed reasonable to me.  If I'd not had 
8 gigs RAM to play with, I'd have probably made it 5 gigs tmpfs, or 
possibly tried 4 and upped it if I needed to.

For a 4-gig physical RAM system or lower, people obviously aren't going 
to have the luxury of using the same reasoning I did.  At 2 and 4 gigs, 
the kernel should be somewhat reasonable at managing things by default, 
tho the lower physical RAM, the more likely people are to find fiddling 
swappiness necessary for optimum results.  If anyone wants to try it with 
a gig RAM, I'd honestly like to see your results, both default and after 
fiddling with swappiness.  It'd make writing the next one of these MUCH 
easier! =:^)  For that matter, altho with two gigs RAM it should still be 
worth it, it'd be nice confirming that, and I'd be quite interested in 
seeing how much fiddling with swappiness was able to improve the results 
there, as well.

Two more things to wrap up.

First, I mentioned striped swap.  It can make a big difference.  If 
anyone has multiple disks and is interested in further details on that, 
just ask.

Second, and really the only place I sort of disagree with Sami, when I 
considered tmpfs here, I decided to go with only one tmpfs, 6 gigs as I 
mentioned, while he chose two, separating his 5 gig PM tmp from his 2 gig 
system tmp.  Perhaps he'll post his reasoning for going separate, but 
here's why I decided on the combined.  Thinking about my usage, I decided 
it wasn't likely that I'd do more than one thing needing substantial 
amounts of temp filespace at once.  Most of the time, that "one thing" is 
portage, but it's occasionally handy to have a tmpfs big enough to say 
copy a DVD image into, to burn it from memory instead of disk, for 
instance.  Standard DVDs run up to 4.7 GB single-sided, and I didn't find 
the idea of ensuring portage wasn't hogging space while I worked with a 
DVD or anything else I might want to store on tmpfs restrictive, so the 5 
gig limit was convenient for that as well.

At the same time, I wanted to limit worst-case maximum tmpfs usage.  
Since I have 8 gigs RAM and it was convenient to do so, I wanted to keep 
total tmpfs usage to some reasonable fraction of RAM, so I'd never be 
forced into swap on account of tmpfs usage alone.  It was thus useful to 
me to create a single tmpfs a reasonable fraction of my memory size, such 
that if something should for some reason start unrestricted writing to 
the tmpfs either due to a buggy program or due to some attempt at a DOS, 
I'd be limiting it to a reasonable fraction of RAM and it could never DOS 
me due to running me out of memory.  (This, BTW, is one reason for the 
"half of physical RAM" tmpfs max size default, as well.)  So I did the 5 
gig thing, added another gig for flexibility while still limiting it to a 
reasonable 3/4 total physical RAM, and called that good.

If I had created two separate tmpfs, the PORTAGE_TMPDIR one and another 
one for use with DVD images or whatever, each five gigs, that would have 
implied a total worst-case tmpfs usage of 10 gigs altho it would have 
allowed me to be stricter on what users/groups I allowed to write to 
each.  I decided the single 6-gig main tmpfs was better for me than two 
smaller ones, maximizing flexibility while more strictly controlling 
maximum system memory usage.

FWIW, I do have a much smaller tmpfs located at the FHS/LSB standard /dev/
shm mountpoint.  However, it is just that, MUCH smaller, limited to 2 MB, 
as standard usage is for quick 0-byte lockfiles, sockets, and other 
similar small files.  If something's abusing that standard and trying to 
store anything of significant size there, I want to know about it right 
away, and a 2 MB size limit should ensure that.  udev is also tmpfs, size 
2 MB, and openrc (with baselayout-2) has its rc-svcdir tmpfs mounted at /
lib64/rc/init.d, size 1 MB, as well.  The total (max) size of these 
"system" tmpfs mounts is thus 5 MB, rather lost in the noise on a multi-
gig memory system.

Again, users with 2 or 4 gigs, let alone a single gig of RAM, will by 
necessity have a somewhat different strategy here than I did, as the 5 
gig baseline for a PORTAGE_TMPDIR tmpfs will by definition be larger than 
their total physical memory size.  However, the idea of restricting worst-
case tmpfs usage remains valid, really even more so, since it WILL be 
running in swap if full.  

OTOH, while multiple tmpfs mounts increases overall resource exposure, 
it /does/ allow one to better restrict access to individual tmpfs 
mounts.  Perhaps that's Sami's strategy.  If he limits writing (and 
perhaps reading too) on his PM temp to the portage user (and root, of 
course), then by separating them, he's limiting exposure on his (assumed 
world writable) 2-gig system tmpfs to its 2-gigs, while I'm exposing that 
whole 6-gig tmpfs to writes by any user.  As I said, perhaps he'll post 
his reasoning and we'll see.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman