Re: [gentoo-server] Server Packages for Gentoo

[BRM]

Kerin -

Thanks for the great point-of-fact. And I honestly have to agree. Portage is certainly the reason I choose Gentoo for all my systems. It's just so much superior to anything else out there - in terms of use only Debian's system even comes close to comparing.

Ben


----- Original Message ----
From: Kerin Millar <kerframil@×××××.com>
To: gentoo-server@l.g.o
Sent: Wednesday, October 1, 2008 6:55:21 AM
Subject: Re: [gentoo-server] Server Packages for Gentoo


2008/9/30 Robert Bridge <robert@××××××××.com>

    On Tue, 30 Sep 2008 09:17:42 -0700 (PDT)
    BRM <bm_witness@×××××.com> wrote:

    > That's a matter of choosing what you install; but that's not specific
    > to Gentoo.
    >
    > MySQL on Gentoo is not going to be any different than MySQL on RHEL
    > or SLES. However, stability - due to differences in versions,
    > patches, etc. - might be different; but should be close to the same.

Or better ...

    Except the Gentoo version will move a lot faster, potentially causing
    problems...

This is potentially true but (a) the term "problems" can be interpreted in different ways (b) it actually cuts both ways (warning: long anecdote follows before leading up to my point) ...

Recently I volunteered some time to help a friend deal with some serious issues he was having in running a popular community site. He'd recently migrated to a dedicated host running CentOS and had assumed that this would address all of the scalability issues he was encountering beforehand. In fact, the situation became worse. When I investigated I discovered that apache/mod_php was running the server into the ground, eventually causing the kernel's OOM killer to spring into action. This situation was not helped by the rather horrid bespoke configuration, with core software having been re-packaged adly by the ISP and effectively held together with rubber-bands and sticky tape. Simply put, it was a complete and utter mess and hopelessly unstable.

Due to the comparitively limited amount of physical RAM and the behaviour exhibited by apache, I suggested that he run lighttpd and php-cgi. I wasn't particularly suprised to find that CentOS did not have official packages and I had to resort to using third-party repositories containing hoplessly outdated packages to find what I needed (or face building from source). I was effectively fighting to be able to make the distro do what I wanted it to do.

After addressing that, he continued to encounter stability issues. I the suggested that he might consider moving to a more flexible distro with a broader range of packages on offer. After learning of the options made available to him by the ISP, the only one that seemed remotely palatable was Debian. He conducted a full re-installation accordingly, and I set up lighttpd, php5-cgi and a number of other components in the stack. Interestingly, not everything he wanted was available - namely the apc opcode cache. Cue messing around installing build-essential and a number of other dependencies manually before having to manually build apc from source.

Anyway, after setting everything up, things seemed to go well initially. But it wasn't before long before disaster struck - after a certain load various php-cgi processes would "run away" and consume inordinate amounts of processor time, with lighttpd unable to service further requests as a result. The only way to address the problem would be to run pkill -9 php5-cgi && pkill lighttpd. Worse, after doing so, the MySQL database that powered his backend would be subtly corrupted - enough to break the bulletin board software at the heart of the site! This would simply happen again and again.

I pursued every angle that I could possibly think of. This is where Debian started to seriously get in my way. I knew that it was a bug, but I hadn't yet identifed which. I wanted to update the key components in the stack to see if the problem had already been addressed. I pinned a newer version of lighttpd from lenny to no avail. I wanted to try a newer version of php but Debian simply does not offer an up-to-date package. Furthermore, it became apparent that "unmasking" (to use a Gentoo-centric term) new software in Debian is very much an all or nothing affair, which is decidedly not what I wanted.

To cut a long story short, I became throughly fed up with the situation and realised that something needed to be done. I therefore conducted a precarious - but ultimately successful - remote migration to Gentoo in-situ and, guess what? After setting up a lean and mean base system and installing lighttpd-1.4.19 and php-5.2.6 fresh out of portage, the site proceeded to work beautifully and without a hitch. And MySQL, which had been a CPU hog on Debian, now runs noticeably more efficiently. Incidentally, after doing a bit more digging I figured out that the system had probably been affected by PHP bug 40286 [1]. At the time of writing, Debian have done nothing about this bug [2] and, I suspect, not a greal deal concerning the 180 or so other bugs that have been fixed upstream in PHP since the 5.2.0 release.

Simply put, Gentoo enabled me to get to where we needed to go - on a fast track to stability no less. And it didn't get in my whilst doing it. In fact, it enabled me to simplify the complexity of the base system to a significant extent through the discriminating employment of USE flags. And, with fantastic components such as openrc/baselayout-2, eselect, webapp-config and, - not least - portage itself, it's a joy to manage.

In actual fact, the components of the base system are _not_ really updated all that often in Gentoo, despite a lot of nonsense that one often hears to the contrary. Since this deployment, there have been 3 minor package updates (one of which was a system package, man-pages) and - what do you know - today a new version of lighttpd is released which fixed 4 security bugs and it's already in the tree. I glanced over the upstream ChangeLog and had no hesitation in applying it to the system in question immediately. Incidentally, I wonder how long it will take the "enterprise" distros to backport the necessary fixes, assuming they even bother at all?

And this leads me up to the point I'm trying to make. There are other distros out there that like to position themselves as the natural choice for sysadmins who seek "stability" or require "enterprise" class packages. They would effectively have you believe that it's viable to run a bunch of frozen packages on a general-purpose system because they are doing the heavy lifting and claim to be backporting the fixes that matter. My view is that this is largely a sham  - there are countless security bugs are never backported, and that's before you even get to the non-security bugs that have a high impact.

Take the kernel for example - it's probably not an exaggeration to say that Gentoo has one of the most pro-actively maintained kernel patchsets around [3] in terms of maintaining branches that upstream like to drop like yesterday's bad news, largely thanks to the combined efforts of the kernel herd on genpatches [4], and the maintainer of hardened-extras. I'd invite anyone who doubts this to take a look at, say, the work that was done on the 2.6.23 branch of hardened-sources [5], above and beyond the related genpatches set, then to compare and contrast with your favourite "enterprise" distro and see exactly how good a job they are doing of looking after your interests. Sure, it's recently been dropped from the tree because we only have the manpower to maintain to maintain so many releases, but it's _still_ probably a far safer kernel than you're getting in the likes of RHEL or Debian! And I'm not even talking about the grsecurity/PaX related stuff here,
 but actual fixes that come from the stable-queue upstream or, in some cases, are not to be found in the stable queue at all (or are not submitted because upstream don't care anymore).

From my perspective, all these distros do is provide the illusion that you are safe in not pro-actively managing your system and completely avoiding the fact of the matter that, yes, there comes a time when software really should be upgraded. For pretty much all of the open-source software that I use on the backend, upgrades typically go very smoothly and fix a heck of a lot more than is ever broken.

Well, this post turned out to be a lot longer than I had anticipated. But I've seen so many comments that allude to Gentoo somehow being unfit for purpose because it doesn't freeze off a so-called "stable" tree so many times that, frankly, I get fed up with it and figured that something had to be said. Gentoo, whilst certainly having its fair share of foibles, doesn't get enough credit for the things that it does well and the things that it does right. If one doesn't like the way that Gentoo does things then there are surely other distros out there that will meet one's expectations, such as they are.

My take: Gentoo is so much more pleasant to manage and administer that I feel like a duck out of water whenever I'm charged with managing anything else. The technology is generally light-years ahead of its contemporaries and I honestly do sleep a lot easier at night knowing that my systems are powered by it. Finally, any extra time expended in managing it is for me (a) well within the margins of what I consider a reasonable amount of effort (b) time well spent (c) produces tangible benefits (more than I could possibly mention here).

Cheers,

--Kerin

[1] http://bugs.php.net/bug.php?id=40286
[2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=431799
[3] http://bugs.gentoo.org/show_bug.cgi?id=185022#c3
[4] http://dev.gentoo.org/~dsd/genpatches/
[5] http://confucius.dh.bytemark.co.uk/~kerin.millar/trunk/2.6.23/