[gentoo-amd64] Re: Why Is Gentoo So Far Behind?

[Duncan]

Frank Peters <frank.peters@×××××××.net> posted
20090615222731.b12f5121.frank.peters@×××××××.net, excerpted below, on 
Mon, 15 Jun 2009 22:27:31 -0400:

> In a lot of cases, for example perl, Xorg, and gcc, the Gentoo
> distribution lags far behind the latest available releases. Even
> allowing the "~amd64" unstable series, this remains true. Why is this
> so?

Others posted their take.  Here's mine.

I don't know enough about perl to have a reasonable opinion there, but I 
besides using them, I follow the Gentoo dev list, and happen to know 
what's going on with the other two.

Consider what a well tested and working gcc means to Gentoo, as opposed 
to what it means for your typical binary distribution.  Really, that 
should be enough right there if you think about it, but let's just put it 
in writing.  In a binary distribution, a gcc that fails to build 
particular packages is no big deal.  In Gentoo, it's a *MAJOR* issue.  In 
a binary distribution, a gcc that gets broken somehow is again, no big 
deal. In Gentoo, it's a *MAJOR* big deal, possibly necessitating a 
complicated recovery procedure involving unpacking a stage tarball 
somewhere and binpkging its gcc, to then install over the broken one on 
the main system.

So, naturally, Gentoo's way more cautious about unmasking a particular 
gcc version.  While micro-version changes (4.3.2 -> 4.3.3) are generally 
bug-fix only anyway and thus get moved thru the process pretty fast, 
normally for minor version changes (4.3 -> 4.4), the process is far 
slower and more complex.  I regularly unmask and test new gcc versions, 
so I know from personal experience what it means and the hassle early 
users go thru.

First, pre-release, weekly snapshots and definitely the -rcs are 
available in the overlays, for anyone wishing to go thru the hassle of 
testing them.  Gentoo devs therefore get a bit of a head-start, knowing 
the trouble areas and often patching troublesome packages to build with 
the new gcc before it's even released.  Then, when it's released 
upstream, the new gcc ebuild is usually in the tree, hard-masked for 
testing, the same day.  Based on the pre-release overlay testing, they 
know the ebuild itself generally works by then, and barring unusual 
issues, gcc itself should compile and install from the ebuild.

However, at release, there's dozens of packages that have yet to be 
updated to be able to actually build with the new gcc.  Many of them 
require patches.  Others require restricting certain CFLAGS or the like.  
A gcc upgrade tracker bug is normally open by then, with a list of all 
the individual package bugs.  As patches are developed and applied (at 
this point, to ~arch packages), the bugs are resolved, and the list of 
unresolved bugs on the tracker shrinks.

Meanwhile, keep in mind that another difference between Gentoo and binary 
distributions is that Gentoo effectively ships often tens, sometimes 
hundreds, of different versions of the same package, due to USE flags.  
Obviously, even testing every package with a late gcc -rc version isn't 
going to catch all the bugs, because each package is effectively many 
slightly different packages, due to USE flag combinations, and it's 
simply impossible to test them all, period.  Thus, after release, users 
like me start testing, and often discover NEW ways in which a particular 
package breaks with the new gcc, filing even MORE bugs to add to the 
tracker, and eventually get fixed.  The binary distributions really have 
it easy in this regard, as they ship one, perhaps two versions of a 
package, with various bits enabled or disabled, take it or leave it, or 
compile it yourself, in which case they do NOT provide support, while 
Gentoo does.

So eventually an ~arch version of most packages has been patched and 
tested to work (under at least some configurations) with the new gcc, and 
it's considered on the way to stable, enough to unmask it to ~arch.

Then the process not quite, but almost, starts over again. Not quite 
because at least they have the list of packages and known ways to fix 
them, now.

Normal package stabilization qualification is 30-days in ~arch without an 
open bug, at which point a package maintainer opens a keyword 
stabilization bug, asking the various archs to test and keyword stable.  
(Note that it's not the package maintainer that marks stable, it's the 
arch devs and their ATs, who test it, marking it stable only if it works 
with an all-stable system config.  Also note that while Gentoo does have  
package.keywords as a method to run a partially unstable system, such a 
system is /not/ tested, and thus is far more likely to run into issues 
than an all-stable or all-unstable system.)

However, with a toolchain package like gcc, the process is far more 
complex than that.  Remember that list of typically a hundred packages or 
more that had to be patched to build with the new gcc? NOW all those 
packages have to make it to stable!  Only when there's a stable version 
of every one of those packages in the list that can build with the new 
gcc version, can that gcc version /itself/ move to stable!

/Now/ you should be /beginning/ to appreciate just how complex the 
journey of a new gcc version is, from pre-release testing in the overlay, 
to release onward masked testing in the tree, to unmasking to ~arch, to 
final stabilization, along with /hundreds/ of new versions of other 
packages with the required patches to build with the new gcc.  That's a 
tough order to follow, and it's really amazing that the process works as 
well as it does.

But, let's also consider what happens from a user perspective when a new 
gcc is setup on the system, and what it might mean if it was done too 
early and broke some packages, therefore requiring keeping both it and 
the older version around.

When a new gcc is installed and people start to use it to compile 
packages, a lot of things can break, until a full emerge --emptytree 
@system @world is performed.  Again, as a new gcc tester, I'm rather well 
acquainted with this.  But if you've done such an upgrade and removed the 
old gcc without doing that --emptytree rebuild, you've probably seen at 
least the most infamous problem, the libstdc++ errors due to all those 
*.la libtool files pointing at the old gcc libstdc++ location instead of 
the new one.  For that there's the fix_libtool_files.sh script, and it 
does a /remarkably/ good job.  However, consider those guys testing those 
weekly gcc snapshots in the overlay!  How many times are they going to 
run fix_libtool_files.sh to point to the new weekly snapshot?

Meanwhile, what about all those packages that break with the new gcc and 
don't have patches applied yet?  Consider for example, kde4.  kde4 
depends on cmake to build.  cmake is gcc version sensitive.  Even when it 
builds against a new gcc, that means it won't work with the old one.  
Thus, while ordinarily if a package won't build with the new gcc, you can 
gcc-config back to the old one and rebuild it that way, that doesn't work 
for kde4, because now you must rebuild cmake, and often several other 
dependencies as well, just to build that one kde4 package that won't 
build with the new gcc.  Then you gcc-config back to the new version, and 
have to rebuild cmake and possibly other dependencies /again/ to continue 
with the other packages.  Sometimes the dependencies just can't be made 
to fit together in a working way, and a user must give up, and remerge a 
significant part of his system with the old gcc again, until a few more 
patches are available.

Of course, particularly those who unmask new gcc versions that aren't 
even in ~arch yet but to some extent ~arch users as well, must get used 
to searching for bugs on packages that won't compile with the new gcc, to 
see if there's a patch there, that hasn't been actually applied to a 
package yet.  Sometimes they have to go looking elsewhere as well, 
Debian, Fedora, etc, for patches that they may have that haven't made it 
to Gentoo bugs, just yet.  All that's a part of running a new gcc 
version, and why it takes so long to get to stable.

Now, all this is enough of a problem for someone who deliberately 
unmasked the new gcc for testing, before it was even ~arch, or even, to a 
lessor extent as there are fewer issues by then, for ~arch users.  But 
consider the protest stable users would (justifiably) raise if they were 
subjected to this!  It's bad enough that if they follow the 
recommendation, they have to emerge --emptytree @system @world.  If they 
choose not to do that, they must at least run fix_libtool_files.sh, or c+
+ apps quit compiling.  And we have enough users (and the bugs they file) 
that don't even know how to do that, or complain about having to do so.  
NOW consider those SAME users having to deal with all the stuff the early 
testers and ~arch users deal with, if a gcc version is keyworded stable 
before its time!

As you can see, life's a LOT more difficult for Gentoo devs when a new 
gcc version comes out than it is for your typical binary distribution.  
If anything, it's actually a wonder that gcc versions get stabilized as 
fast as they do, and that we're not still back on gcc-3.something.

The problem with xorg is rather different, and much simpler to explain.  
Basically, it usually comes down to all those proprietaryware graphics 
drivers users.  As with most distributions, Gentoo is hesitant to 
stabilize a particular xorg until the users of the various graphics cards 
using proprietaryware drivers have a supported graphics driver they can 
upgrade to as well.  As the quote in my sig describes, the 
proprietaryware driver vendors really do in this case serve as xorg 
masters, holding it back otherwise unnecessarily.

Then for xorg in particular, there's the whole configuration change thing 
going.  Before a version of xorg with major config changes goes stable, 
an upgrade guide must be created, detailing how to deal with the config 
changes along with any other issues that arise.  As others have 
mentioned, docs, formerly an area where Gentoo shined and still an area 
where they're reasonably good (how many other distributions cover upgrade 
config changes in such detail, not many!), has been dragging a bit 
behind, lately.

Finally, because X is such a major package, there's also a bit of the 
same issues to deal with that I described for gcc above, but nowhere near 
to the same level, and at least for stable, as I said, it's very often 
the proprietaryware driver vendors holding things up, and when it's not, 
it's often the development of the upgrade docs.

> I had first considered moving to Gentoo in the fall of 2008, but after
> noticing that the only version of gcc available at that time was
> gcc-3.x, I postponed the change.  In the spring of 2009, Gentoo finally
> moved up to gcc-4.3.x and then I made the transition.  But the update to
> the 4.3 series was a long time in coming.

The above covers that in great detail.  Also, again as others have noted 
and as I alluded to above, there are in fact snapshot and -rc ebuilds 
available from long before a gcc version is ever released.  But for the 
reasons covered above, they aren't in the tree prerelease, and when they 
do hit the tree at upstream release, they're first hard masked, then 
unmasked to ~arch, thru an extremely complex process that takes quite 
sometime.

But they're certainly available for those brave/hardy/stupid enough to 
try them!
 
> The latest perl, released some time ago, is version 5.10 but Gentoo
> includes only 5.8.8.

As I said, no comment there, as I simply don't know enough about it to 
have an opinion, educated or not.

> The latest Xorg has restructured certain libxcb dependencies, which has
> caused a lot of problems for a lot of packages, and Gentoo is behind
> these changes as well.
>
> (Ironically, it was this libxcb issue as well as the whole Xorg
> modularity mess that first motivated me to seek out Gentoo.)

Actually, libxcb was in fact one of the gcc-like complexity issues of 
xorg.  In this case, previous versions had a particular library that many 
X based packages built against, that was no longer available in the newer 
version.  For those running LDFLAGS=--as-needed (as I do), the problem 
wasn't all that bad.  However, for those NOT running with that flag 
(which does break some packages, and used to break a LOT more, before 
flameeyes started working on the problem), the easiest way around the 
problem was to recompile everything that broke -- in a specific order 
that portage didn't always manage properly -- and there were a LOT of 
package in this broken list.

So, stabilization (and indeed, the move from the overlay to the tree and 
~arch) was delayed while they worked out and tested a solution.  There's 
now a script that can be run that takes care of most of the problems, 
without a recompile.  Again, it's a libtool *.la file issue.  Then that 
solution had to be documented...

Yes, some binary-only distributions beat us to it, but again, all they 
have to worry about is getting the compilation correct on the single 
version of each package they normally provide (for each arch).  They 
don't have to worry about the recompile issue that was Gentoo's big 
holdup here for xorg/libxcb, or pretty much all the time for gcc.

> Now I am not actually voicing a complaint.  Gentoo, IMO, is still the
> best distribution for Linux.  I am just wondering why there is such a
> great lag before a package version is deemed stable -- or even unstable.
>  In my experience with maintaining my own Linux system, I never had any
> great issues with always installing the latest "bleeding" edge software.

I'm glad you're not complaining, and actually, I understand why someone 
not acquainted with the particular issues a from-source distribution 
deals with, particularly a distribution used by and depended on by as 
many people as Gentoo... I can well understand how they might find that 
Gentoo looks slower on the uptake than it should be.

But once you understand what's going on behind the scenes, and how much 
work Gentoo devs, ATs and ~arch users put in to ensure a particular 
package is really stable before it's keyworded that way (and why even 
with all that work sometimes things slip thru the cracks), the mystery 
then becomes much more one of how in the world it all gets done, and how 
in the world it continues to function as well as it does!

Of course, if Gentoo were to even do something as simple as do away with 
most USE flags, cutting the number of possible package variables 
accordingly, it would go a HUGE way toward eliminating all that extra 
work and all those extra bugs from all the variations.  Similarly, Gentoo 
could do away with the CFLAGS etc settings, or just filter them for most 
packages, again cutting down on the number of bugs tremendously.  
Gentoo's offering would then be far closer to what the binary 
distributions offer.  Then, since we'd have just done away with pretty 
much the entire set of advantages of building from source, we could do 
away with the from-source entirely, and just ship the binaries, bringing 
us into pretty much direct comparison with the binary distributions.

Of course, that would kill the whole reason Gentoo's so great to run for 
many of us.  We LIKE it's customizability, even if it means thinner 
testing and thus dealing with a few extra bugs and longer stabilization 
cycles, and even if it does mean waiting a bit for packages to compile!

Meanwhile, it should also be noted that "in my experience with 
maintaining my own Linux system", doesn't quite compare to dealing with 
THOUSANDS of people running one or more systems each, many with entirely 
DIFFERENT configurations, and the headaches that can mean, trying to get 
something working on ALL of them at once, without breaking the obscure 
and rather strange configuration that guy off in the corner with even 
more customization than usual (umm... that'd be me! see the make.conf I 
posted a few days ago for just one example of "strange" and "even more 
customized than usual" =:^) is running.

Yes, that means even if that system one was maintaining was a Linux from 
Scratch system.  (I've not done Linux from Scratch yet and probably never 
will, because Gentoo's the perfect balance of customization and 
automation for me and I discovered Gentoo before Linux from Scratch, but 
a lot of Gentoo users have.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman