Gentoo Archives: gentoo-amd64

From: Duncan <1i5t5.duncan@×××.net>
To: gentoo-amd64@l.g.o
Subject: [gentoo-amd64] Re: Symlinks vs. Bind mounts.
Date: Tue, 12 Aug 2008 14:02:15
Message-Id: pan.2008.08.12.14.02.06@cox.net
In Reply to: Re: [gentoo-amd64] Re: Symlinks vs. Bind mounts. by Beso
1 Beso <givemesugarr@×××××.com> posted
2 d257c3560808120123t4733d15pc81f4968cbba9bdf@××××××××××.com, excerpted
3 below, on Tue, 12 Aug 2008 08:23:06 +0000:
4
5 > 2008/8/12 Morgan Wesström <gentoo-amd64@×××××××××.biz>
6 >
7 >> Duncan wrote:
8 >>
9 >>> Now, if you /really/ want to make a difference in portage's speed,
10 >>> consider pointing PORTAGE_TMPDIR at a tmpfs.
11 >>>
12 >> This advice caught my attention since I moved my tmp space to Reiserfs
13 >> for performance reasons. My knowledge of tmpfs is limited but I think
14 >> it is a filesystem that uses RAM and can grow and shrink dynamically,
15 >> right? If I follow this advice, what happens when I compile something
16 >> like Open Office which allocates 3-4GB in /var/tmp during compilation
17 >> and I only have 2GB physical RAM in the computer?
18 >>
19 > you'll use swap partition. but you'll not allocate all that ram space
20 > with openoffice. i've tried to compile it twice. first time it was on
21 > disk and it took almost 14 hours of compilation. the second time was on
22 > tmpfs with 3.8gb and a 6gb swap file and it took less than 8 hours and
23 > i've never filled the swap partition. to put at maximum use this method
24 > with low ram then don't start xorg and graphical terminals but only the
25 > base vt and compile from there. this will save you quite some ram space.
26 > you'll also find that the -j option in your make.conf could be increased
27 > much when going with tmpfs. for example i've passed from -j5 in hd use
28 > to -j9 in tmpfs and i still have a very good and usable graphical
29 > system. with the old single core pc i was using -j2 in hd and -j5 on
30 > tmpfs. this dramattically decreases compilation time.
31
32 This well illustrates the speed-up of PORTAGE_TMPDIR on tmpfs. Now, I
33 don't use -j -l MAKEOPTS to control load so much as to control memory
34 usage. I can have a load in the hundreds and it doesn't matter that
35 much, particularly with per-user scheduling turned on in the kernel, but
36 of course memory can still be an issue if compiling into tmpfs and too
37 many jobs get going.
38
39 I'll second that tmpfs is swappable as well. That makes it great for
40 compiling (as long as you have enough swap), because even when real
41 memory isn't enough to hold it all and it swaps some of it, many of the
42 shortest lived files never hit disk at all, making it MUCH faster, and,
43 as Beso points out, MUCH more responsive on other things as well, even
44 with more jobs going, because of the nature of disk I/O.
45
46 Morgan also asks (down-thread) about compiling OOo with 2 gig RAM and 1
47 gig swap. Yes, that'll be problematic. One thing you could do would be
48 to shrink or kill entirely the partition you are using for /tmp and/or
49 /var/tmp (or wherever else you are currently compiling OOo into that
50 obviously has that much space) and turn it into more swap. Another
51 alternative is to setup swap files. With 3 gigs total mem+swap, if you
52 don't want any more swap under normal conditions, this would allow you to
53 create it temporarily out of normal filesystem space, only for merging
54 big stuff like OOo. It won't be quite as fast that way, but it should
55 still be way faster than compiling it to PORTAGE_TMPDIR on disk. A third
56 alternative would be to point PORTAGE_TMPDIR at tmpfs for normal merges,
57 and just point it back at disk temporarily when emerging OOo. Of course,
58 that would be slow again, but no slower than now, and you'd still have
59 the speedup when merging most stuff.
60
61 Personally, I'd recommend the first, recovering disk space currently used
62 for /tmp or /var/tmp, and turning it into swap.
63
64 > @duncan: do you remember that some time ago you were speaking about
65 > posting the scripts to compile the kernel with the make.conf
66 > optimizations, but you haven't posted them any more. do you still use
67 > them?!
68
69 I remember, and yes, I'm still using them. I had to fix something
70 recently, when debianutils changed its kernel install scripts, but I
71 still use them.
72
73 I still have to get it on the webspace, but thanks for reminding me.
74
75 --
76 Duncan - List replies preferred. No HTML msgs.
77 "Every nonfree program has a lord, a master --
78 and if you use the program, he is your master." Richard Stallman