[gentoo-trustees] How are we going to spend money?

[Kurt Lieber]

All --

We haven't yet discussed how we're going to spend money (or decide how to
spend money) but below is a message from Robin that I think clearly
justifies spending some money on lark.  It might be a good catalyst to get
us to figure out what our budgeting and expenditure approval process is
going to be.  It can be as simple as someone (like me) posting an email
(like this) to this list or it can be somethimg more complex.

Ideas?

--kurt

----- Forwarded message from "Robin H. Johnson" <robbat2@g.o> -----

Date: Sat, 22 May 2004 02:55:28 -0700
From: "Robin H. Johnson" <robbat2@g.o>
Subject: CVS performance improvements
To: Gentoo Infrastructure <gentoo-infrastructure@l.g.o>

Some weeks ago, in #-infra we had a discussion about improving the
performance of the CVS server.

Some months ago we already moved the CVS lockfiles to a tmpfs which provided a
very good speedup for many operations, but still left room for improvement.
Before the lockfile move, a full cvs up could take more than 45 minutes if the
server was very busy. The lockfile move got us down below 20 minutes the great
majority of the time. As the server has become busier, we've recently been
hitting 20 minutes as the average time for a cvs up again.

Outside of the CVS tree itself, CVS uses two things:
lockfiles - (detailed above)
tmpfiles - stored in /tmp/cvs*/ by default, CVS figures out what it will
send you by building stuff here - a mixture of what you upload to it on
your file status and the latest file status.

The contents of tmpfiles don't last much longer than the duration of a
single checkout, so it's suited for caching in memory, but there is a
problem with this. It is a LOT of files. It's usual space allocation is
~4 very small files (50-200 bytes ea.) for each directory - approx 1kb
of actual data per directory. We have 18812 directories for our
gentoo-x86 module presently, so this is a LOT of wasted space when using
4kb blocks. Checkouts are presently all on disk, and use up this space
already when they occur. ~300mb for a single 'cvs co gentoo-x86'.
However due to the large number of files, esp. when we have multiple
checkouts/updates going, performance really hurts on /tmp (which is on
our system drive).

Before when were wondering how to solve this bottleneck, we considered added
more ram and using tmpfs. tmpfs is locked at using 4kb blocks due to how it's
designed, so we'd need huge amounts of memory - we frequently have 5-8
simultaneous large updates going (5*300mb = 1.5gb).

As a quick profiling of CVS usage, I processed the some logs for the last 14
days, and excluded the gmirror, gweb and pylon users as they look to be using
scripts extremely and throwing off some statistics.

Top 5 users:
Total Runs | Username
---------------------
1350  	   | gweb
1382  	   | vapier
1549  	   | mr_bones_
1997  	   | gmirror
2765  	   | pylon

UTC  | Average CVS
Hour | runs per hour
---------------------
00h  |  72.571
01h  |  62.357
02h  |  53.786
03h  | 245.786
04h  |  51.357
05h  |  63.857
06h  |  43.214
07h  |  56.786
08h  |  47.786
09h  |  46.714
10h  |  42.786
11h  |  50.643
12h  |  62.714
13h  |  58.643
14h  |  73.000
15h  |  77.357
16h  |  69.500
17h  |  82.571
18h  | 117.714
19h  |  66.643
20h  |  75.857
21h  |  65.143
22h  |  66.143
23h  |  77.857

In the above numbers, ~200 of the daily items are from pylon all around 3am.
gweb and gmirror together account for 10 runs / hour.  Without pylon, gweb,
gmirror, we have ~1300 developer-initiated CVS runs each day. (I don't see any
other obvious script accesses). Of that, some 20% are cvs updates.

Looking to cut down on the wasted tail space, I ran some more tests on lark
today. I originally wanted to use ramdisk as a module (so I can specify the
size to 128mb), but this isn't possible as we have ramdisk compiled in for
initrd.  Instead, I've utilized some of the MTD support in the kernel (all
added as modules), which provides a similar functionality to a ramdisk.
I loaded the MTD as a 64mb chunk in memory, and formatted with reiserfs, so I
could try out tail-packing for space saving.

A full 'cvs co gentoo-x86' needs 20mb of space with. An a full update of your
tree if you are very out of date (a month or more) could use ~50mb of space.
A single cvs checkin won't take more than 100kb absolute worst case.

I believe the MTD+reiserfs route is definetly much more cost-effective on RAM
than tmpfs when the system is busy, but it has the disadvantage that when the
system isn't busy, the kernel would have reclaimed the tmpfs space, but can't
do so with the MTD space.

Results from a testing run of MTD+reiserfs running on my user only:
(did 3 cvs up on month-old repos, and 2 full cvs co)
- 5-8 minutes for cvs up/co !! This is a two to three times faster than before,
and it won't increase significently in future, or with large numbers of users,
as memory has constant access time :-).

For this testing, I had 8 other developers doing either cvs up or cvs co at the
same time, and the i/o usage pushed loadavg to ~15 on lark.

There is one remaing problem, namely I can't get MTD to provide me with a block
larger than 64mb, or multiple blocks that span beyond 64mb. I'll play with some
of the MTD stuff at work this week (on a box with lots of memory), and see what
is needed to use lots of memory with MTD.

Originally when discussing tmpfs for the tempfiles and looking at the
purchasing of more ram, we said 2gb would be just enough, but thanks to
MTD+Reiserfs, it looks like 1gb will do very nicely, 2gb if we think we'll be
on lark for more than another 2 years (we could always add more memory later
when we get to that point).

-- 
Robin Hugh Johnson
E-Mail     : robbat2@××××××××××××××.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85



----- End forwarded message -----