[gentoo-amd64] Re: Fragmentation

[Duncan]

Peter Humphrey posted <43F20505.3040207@××××××××××.uk>, excerpted below,
on Tue, 14 Feb 2006 16:27:49 +0000:

> Gavin Seddon wrote:
>> From reading these posts I have sorted out 'other' issues that are off
>> list.  Namely [...] creating a partition for /usr/portage to 'aid'
>> fragmentation.
>
> I did that some time ago in a simple-minded fashion, but I've had to
> revise my layout somewhat. I had an ext3 partition solely for
> /usr/portage, and it was mounted on that node, but every emerge --sync
> deleted the /lost+found directory. I don't know how serious that is, but
> of course no-one likes to have damaged file systems on their boxes, so I
> used tune2fs -c to set the mount count to 1 so that it was repaired
> every time the box booted, but that was taking too long and it was only
> a palliative anyway. The solution was easy, though of course I didn't
> see it at first! I only had to point /usr/portage to /usr-bits/portage
> (/usr-bits being the mount point of the partition) instead of mounting
> the partition directly in place.

If you have /usr/portage (or more precisely, your portage tree, wherever
it exists on your file system, since it can be moved anywhere and the
pointer in make.conf changed accordingly) on its own partition, consider
making it reiserfs, even if you don't consider reiserfs stable enough for
regular use.  Correspondingly, those wanting a "safe" but relatively
high-use location for testing reiser4 should fine the portage tree a very
good choice.

Here's the logic: The portage tree (without the packages subdir if you use
FEATURES=buildpkg, and with or without the distdir package sources, your
call) has two very important characteristics that make it a perfect match
for either reiser file system, it has a very high number of very small
files -- less than a filesystem block -- and the data in it is ultimately
backed up -- available from multiple sources on the net, so easily
redownloadable should anything go wrong, thus addressing the distrust
issue -- some folks don't consider reiserfs stable enough to store
critical data on, and I myself wouldn't consider reiser4 stable enough for
non-redundant critical data.

When small files are stored on a regular filesystem, including ext2/3,
they are stored by block, each file taking up a full filesystem block of
data regardless of whether it's a single byte or exactly a block.
Likewise, a file a block and a byte long will take up two blocks worth of
space.

Reiserfs has tail-packing, altho it can be turned off.  It stores the
<1-block ends of files (the entire file if it's less than a block, the
remainder of the file if it's more than a block but not an exact number of
blocks) packed together, requiring far less space.  The savings can be
greater than 50% if the data is all small files.  That is, while a regular
filesystem will require more than twice the actual data space to store a
set of small files, two gigs to store a gig of file data, reiser will
require only the single gig to store that same gig of data (plus the
journal space, but that's there with any journaled filesystem, including
ext3, and of course the metadata storage, that is, the inodes).

Likewise, reiserfs has been optimized to make working with small files
fast (in case you are wondering, the parallel for large files is xfs), and
there are a number of reports floating around on the forums of folks that
have switched their portage tree to reiserfs and been shocked at just how
much faster emerge --pretend and similar operations turned out to be.
Being conservative and because I've never had my portage tree on anything
else, so I have no first-hand experience on other filesystems to compare
to, I'll only say it shouldn't be /slower/, that is, if those operations
take longer on reiserfs, something's definitely wrong, but I'm not going
to claim any speedup, only that the storage efficiency is higher,
space-wise, and that it isn't slower to access.

As I said, some folks are concerned with reiserfs' reputation for
instability.  I haven't found that to be the case since the kernel
defaulted to journal=ordered for reiserfs, but in any case, as long as
it's stable enough to keep temporary data on (and it is certainly that),
that shouldn't be an issue for the portage tree, since recovery is only an
emerge sync away.  With an exception for those with infrequent or very
expensive per-byte or very slow (analog dialup) internet connections, who
probably aren't going to be using Gentoo in any case, there's therefore no
data stability issues with the portage tree on reiserfs, even for those
that wouldn't trust it with their regular data.

That makes reiserfs the best choice for the portage tree, where the
portage tree is on its own partition, anyway.  Those who don't trust
reiserfs for data stability should have no qualms here, because the data
is ultimately backed up in any case, and reiserfs /will/ be /far/ more
efficient at storing the tree, and /likely/ will be faster, as well, altho
I can't personally verify that as I've never run the tree on anything else
to compare speed against.

As I said, you may want to keep the packages dir, /usr/portage/packages by
default, on another partition.  This is easiest to accomplish simply by
pointing it elsewhere in make.conf.  The distdir subdir isn't synced with
the portage tree, but contains a local cache of source tarballs that
portage has downloaded for various merges.  As such, it's ultimately
backed up to the internet as well, but because those tarballs are fetched
by portage one at a time as it needs them, not synced with the tree, and
because the files aren't as small in any case, some folks might want to
keep this separate from the portage tree as well, tho the urgency isn't as
great here as it would be with packages.

All that dealt with, there's only one possibly valid reason I'm aware of,
for those already splitting out the portage tree onto its own partition,
why they might /not/ wish to use reiserfs.  Those that only have ext2/3
configured in their kernel may not wish to bother configuring reiserfs for
just the portage tree.  I'm actually in that situation with ext2/3 -- I
don't have anything on my system using it, so there'd have to be a
stronger than usual reason to use it on a particular partition, in ordered
to justify the bother of compiling it into the tree.

As for the lost&found dir, as someone else mentioned, that's trivial.  The
only use for it is when the filesystem finds something during an fsck that
isn't properly linked, that might still be needed.  fsck creates
lost&found in the root dir of the filesystem to place these lost files in,
as it finds them.  It's recreated if needed, so IMO it's actually better
/not/ to have a lost&found by default, as that way, if it exists, you know
to look in it and see what fsck might have recovered, and either delete it
or move it back into its usual place in the tree.  Again, because the
entire portage tree is recoverable from the net with an emerge sync
anyway, it should be entirely safe to ignore a lost&found and have it
deleted in a sync, in any case, because the data will be updated with an
emerge sync anyway, and it's less hassle to do that than to manually
figure out what any files there might be and where they go in the tree,
when even if you did, an emerge sync might simply be deleting the file
anyway, as outdated.

> A word of caution for anyone considering adopting Duncan's scheme
> without much thought: what he says is certainly good sense, but don't go
> copying him if you're just splitting out bits of the file system that
> can be made common to different running systems. I spent half of last
> night exploring some of the snags! My aim was to separate some large
> slabs of files into their own partitions and mount those partitions on
> whichever system I was booting. I have four Linux systems multibooting
> on this box, and it seems tidy to find common areas and treat them as
> such.
>
> Don't combine systems' /var/log directories - you will end up with
> deeply troubled emerge.log and PORT_LOGDIR records.

"Deeply troubled" is an apt description. <g>  In any case, combining
/var/log dirs makes no sense, because what's the log info there for if not
to be able to examine should it be necessary for troubleshooting or record
keeping purposes?  Throwing the logs from multiple independent boot
systems into the same location, with no way to tell what belongs to what
system, can only confuse things, and destroys the entire supposition
behind having the logs in the first place.  (Note that this is different
than the convenience of running a central syslog with multiple machines
logging to it, because in that case, the log will have machine
identification labels to sort out which logged events correspond to which
system.  Just using the same partition for everything just jumbles
everything up in a big mess, defeating the purpose of logging in the first
place!)

> Don't combine systems' /usr/src directories. It won't do much harm, but
> the records of which kernel version is installed in each system will
> cause overwriting anyway, thus spoiling the idea, especially if you have
> USE=symlink for gentoo-sources.

I'd rather say "Know what you're doing if you combine /usr/src dirs."  It
can be done if the appropriate organization is maintained, but as you
point out, automating the /usr/src/linux symlink with USE=symlink for the
kernel-sources packages is NOT a good idea if you are running a combined
/usr/src.  Perhaps a more intelligent solution could be borrowed from
Mandrake (and I suppose Mandriva continues the idea, but don't know),
where an initscript setup a number of symlinks, and this one could be
included.  The idea being to set the symlink to point to the sources
corresponding to the kernel booted, where that is possible, leaving it
alone if there are no sources found that correspond to the booting kernel.

So... /usr/src can be multi-boot combined, but it's not as trivial as one
might expect if one doesn't consider the consequences, so don't do it
without some thought, first.

Actually, that "don't do it without some thought, first" can be applied to
a /lot/ of thing! =8^)

> I'm not yet sure of the wisdom of combining /var/tmp from different
> systems: I haven't yet sorted out the consequences for the portage work
> directories. Watch this space.

By definition, /tmp and /var/tmp should be multi-boot combineable, and
combineable between the two, as well (my /var/tmp is simply a symlink to
/tmp, altho on a multi-human-user system, there are security issues one
should consider before doing it -- yet another place to "don't do it
without some thought, first" <g>), because the data is by definition
"temporary", which in this case is defined as "not needing to survive a
reboot".

That "tmp" is defined as "not needing to survive a reboot" is, BTW, the
official FHS (File Hierarchy Standard, part of LSB aka Linux Standard
Base) definition as well, AFAIK -- the idea being that the practice of
certain distributions, deleting everything in /tmp and /var/tmp at boot,
is specifically allowed and shouldn't cause any malfunctions.

In any case, there's no damage to portage by combining those dirs, or
removing the contents at boot, either.  If you have something set up
locally that saves data across reboots to either /var/tmp or /tmp,
consider changing it, as that's not what those dirs are for, and expecting
them to be safe for that could get broken at some point.

> Apologies if I'm not making much sense today - blame the loss of sleep
> and the head cold that probably caused it :-(

Actually, great sense!  Thanks for bringing up the possibility of
multi-boot partition combines!  It certainly adds to the information
available in the discussion!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-amd64@g.o mailing list