Re: [gentoo-server] Server Packages for Gentoo

[Kerin Millar]

2008/9/30 Robert Bridge <robert@××××××××.com>

    On Tue, 30 Sep 2008 09:17:42 -0700 (PDT)
    BRM <bm_witness@×××××.com> wrote:

    > That's a matter of choosing what you install; but that's not specific
    > to Gentoo.
    >
    > MySQL on Gentoo is not going to be any different than MySQL on RHEL
    > or SLES. However, stability - due to differences in versions,
    > patches, etc. - might be different; but should be close to the same.

Or better ...

    Except the Gentoo version will move a lot faster, potentially causing
    problems...

This is potentially true but (a) the term "problems" can be interpreted in
different ways (b) it actually cuts both ways (warning: long anecdote
follows before leading up to my point) ...

Recently I volunteered some time to help a friend deal with some serious
issues he was having in running a popular community site. He'd recently
migrated to a dedicated host running CentOS and had assumed that this would
address all of the scalability issues he was encountering beforehand. In
fact, the situation became worse. When I investigated I discovered that
apache/mod_php was running the server into the ground, eventually causing
the kernel's OOM killer to spring into action. This situation was not helped
by the rather horrid bespoke configuration, with core software having been
re-packaged adly by the ISP and effectively held together with rubber-bands
and sticky tape. Simply put, it was a complete and utter mess and hopelessly
unstable.

Due to the comparitively limited amount of physical RAM and the behaviour
exhibited by apache, I suggested that he run lighttpd and php-cgi. I wasn't
particularly suprised to find that CentOS did not have official packages and
I had to resort to using third-party repositories containing hoplessly
outdated packages to find what I needed (or face building from source). I
was effectively fighting to be able to make the distro do what I wanted it
to do.

After addressing that, he continued to encounter stability issues. I the
suggested that he might consider moving to a more flexible distro with a
broader range of packages on offer. After learning of the options made
available to him by the ISP, the only one that seemed remotely palatable was
Debian. He conducted a full re-installation accordingly, and I set up
lighttpd, php5-cgi and a number of other components in the stack.
Interestingly, not everything he wanted was available - namely the apc
opcode cache. Cue messing around installing build-essential and a number of
other dependencies manually before having to manually build apc from source.

Anyway, after setting everything up, things seemed to go well initially. But
it wasn't before long before disaster struck - after a certain load various
php-cgi processes would "run away" and consume inordinate amounts of
processor time, with lighttpd unable to service further requests as a
result. The only way to address the problem would be to run pkill -9
php5-cgi && pkill lighttpd. Worse, after doing so, the MySQL database that
powered his backend would be subtly corrupted - enough to break the bulletin
board software at the heart of the site! This would simply happen again and
again.

I pursued every angle that I could possibly think of. This is where Debian
started to seriously get in my way. I knew that it was a bug, but I hadn't
yet identifed which. I wanted to update the key components in the stack to
see if the problem had already been addressed. I pinned a newer version of
lighttpd from lenny to no avail. I wanted to try a newer version of php but
Debian simply does not offer an up-to-date package. Furthermore, it became
apparent that "unmasking" (to use a Gentoo-centric term) new software in
Debian is very much an all or nothing affair, which is decidedly not what I
wanted.

To cut a long story short, I became throughly fed up with the situation and
realised that something needed to be done. I therefore conducted a
precarious - but ultimately successful - remote migration to Gentoo in-situ
and, guess what? After setting up a lean and mean base system and installing
lighttpd-1.4.19 and php-5.2.6 fresh out of portage, the site proceeded to
work beautifully and without a hitch. And MySQL, which had been a CPU hog on
Debian, now runs noticeably more efficiently. Incidentally, after doing a
bit more digging I figured out that the system had probably been affected by
PHP bug 40286 [1]. At the time of writing, Debian have done nothing about
this bug [2] and, I suspect, not a greal deal concerning the 180 or so other
bugs that have been fixed upstream in PHP since the 5.2.0 release.

Simply put, Gentoo enabled me to get to where we needed to go - on a fast
track to stability no less. And it didn't get in my whilst doing it. In
fact, it enabled me to simplify the complexity of the base system to a
significant extent through the discriminating employment of USE flags. And,
with fantastic components such as openrc/baselayout-2, eselect,
webapp-config and, - not least - portage itself, it's a joy to manage.

In actual fact, the components of the base system are _not_ really updated
all that often in Gentoo, despite a lot of nonsense that one often hears to
the contrary. Since this deployment, there have been 3 minor package updates
(one of which was a system package, man-pages) and - what do you know -
today a new version of lighttpd is released which fixed 4 security bugs and
it's already in the tree. I glanced over the upstream ChangeLog and had no
hesitation in applying it to the system in question immediately.
Incidentally, I wonder how long it will take the "enterprise" distros to
backport the necessary fixes, assuming they even bother at all?

And this leads me up to the point I'm trying to make. There are other
distros out there that like to position themselves as the natural choice for
sysadmins who seek "stability" or require "enterprise" class packages. They
would effectively have you believe that it's viable to run a bunch of frozen
packages on a general-purpose system because they are doing the heavy
lifting and claim to be backporting the fixes that matter. My view is that
this is largely a sham  - there are countless security bugs are never
backported, and that's before you even get to the non-security bugs that
have a high impact.

Take the kernel for example - it's probably not an exaggeration to say that
Gentoo has one of the most pro-actively maintained kernel patchsets around
[3] in terms of maintaining branches that upstream like to drop like
yesterday's bad news, largely thanks to the combined efforts of the kernel
herd on genpatches [4], and the maintainer of hardened-extras. I'd invite
anyone who doubts this to take a look at, say, the work that was done on the
2.6.23 branch of hardened-sources [5], above and beyond the related
genpatches set, then to compare and contrast with your favourite
"enterprise" distro and see exactly how good a job they are doing of looking
after your interests. Sure, it's recently been dropped from the tree because
we only have the manpower to maintain to maintain so many releases, but it's
_still_ probably a far safer kernel than you're getting in the likes of RHEL
or Debian! And I'm not even talking about the grsecurity/PaX related stuff
here, but actual fixes that come from the stable-queue upstream or, in some
cases, are not to be found in the stable queue at all (or are not submitted
because upstream don't care anymore).

From my perspective, all these distros do is provide the illusion that you
are safe in not pro-actively managing your system and completely avoiding
the fact of the matter that, yes, there comes a time when software really
should be upgraded. For pretty much all of the open-source software that I
use on the backend, upgrades typically go very smoothly and fix a heck of a
lot more than is ever broken.

Well, this post turned out to be a lot longer than I had anticipated. But
I've seen so many comments that allude to Gentoo somehow being unfit for
purpose because it doesn't freeze off a so-called "stable" tree so many
times that, frankly, I get fed up with it and figured that something had to
be said. Gentoo, whilst certainly having its fair share of foibles, doesn't
get enough credit for the things that it does well and the things that it
does right. If one doesn't like the way that Gentoo does things then there
are surely other distros out there that will meet one's expectations, such
as they are.

My take: Gentoo is so much more pleasant to manage and administer that I
feel like a duck out of water whenever I'm charged with managing anything
else. The technology is generally light-years ahead of its contemporaries
and I honestly do sleep a lot easier at night knowing that my systems are
powered by it. Finally, any extra time expended in managing it is for me (a)
well within the margins of what I consider a reasonable amount of effort (b)
time well spent (c) produces tangible benefits (more than I could possibly
mention here).

Cheers,

--Kerin

[1] http://bugs.php.net/bug.php?id=40286
[2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=431799
[3] http://bugs.gentoo.org/show_bug.cgi?id=185022#c3
[4] http://dev.gentoo.org/~dsd/genpatches/
[5] http://confucius.dh.bytemark.co.uk/~kerin.millar/trunk/2.6.23/