[gentoo-amd64] Re: kdelibs insanity

[Duncan]

Frank Peters <frank.peters@×××××××.net> posted
20090731010923.8875a3af.frank.peters@×××××××.net, excerpted below, on 
Fri, 31 Jul 2009 01:09:23 -0400:

> It should also be considered that perhaps the upstream developers are
> making things more difficult.

No kidding!

> Many new packages that have been released
> seem to break everything that depends on them.  For example, a new jpeg
> library, version 7, was released recently (which has not yet made it
> into Gentoo) that will require a rebuild of every program that processes
> images, and this includes the extensive GTK package as it uses libjpeg
> for its pixbuffer loader.

Thanks for the warning!  I hadn't read anything about that one yet.

> Another example is the new gcc compiler,
> version 4.4.x.  I have noticed that many packages will fail to compile
> with the new version of gcc and this necessitates that the previous
> version, 4.3.3, be also kept installed on the system.

I routinely install any new gcc version before it's unmasked to ~arch.  
As such, I'm used to dealing with packages that won't yet compile with 
the new version, searching Gentoo and other bug systems for patches, 
etc.  Due to the hard work of many even before a public release (Gentoo 
toolchain folks and adventurous users try rebuilding with the rcs, and 
some routinely test weekly snapshots during development.  Debian as well 
tests, and many of our patches to make existing packages compile with new 
gcc versions come from there.  Fedora Rawhide, similarly, and I've 
actually worked with SuSE devs on testing a patch for an upstream package 
(pan) that I'm involved with upstream, myself.

But as we're talking about Gentoo tools making it easier, slotted gcc and 
gcc-config definitely belong on that list, as they SURE make it easier 
testing various new compilers, and switching back to the old one when 
necessary.  The process would be SERIOUSLY harder without this sort of 
Gentoo tools and slotted gcc.

> Lastly, do I need
> to mention the fiasco with the update of the xcb package for X Window?
> Once again, a single change has broken everything that has gone before.

That one actually had FAR less effect on me than I expected.  From the 
Gentoo/X project discussion of things (in the tracking bug and on the 
desktop list, IIRC, the former discovered by reading the latter), I 
expected something on the size of the expat fiasco, but with virtually 
all X apps needing recompiled.  Never-the-less, I upgraded (again, while 
it was masked, actually, I grabbed it while it was still on the X 
overlay, IIRC), and things weren't anywhere NEAR that bad.  I think I had 
a half dozen, maybe a dozen packages to upgrade, instead of the couple 
hundred or so I expected.

But I expect what saved me on that was having -as-needed in LDFLAGS.  It 
really DOES make a HUGE difference.  The goal is eventually to make that 
the Gentoo default, but there's still a lot of packages that don't work 
with it, tho most of them have been patched or at least have filter-
ldflags called for it, so it doesn't hit users as much as it used to.  
Flame-eyes is the expert on -as-needed (it was his blog I discovered it 
on), and his tinder-box has really weeded out a bunch of issues both 
there and elsewhere.  That alone is a huge benefit to Gentoo, even if he 
didn't contribute huge amounts of fixes and technical help for other 
devs, which he does.  (Unfortunately, he reminds me a lot of my father 
and sister.  All three are extremely bright, genius level or close to it, 
all three workaholics -- they see so much to be done and so few able to 
do it, and try to do it all themselves!  And all three have the health 
problems that go with the territory.  All three of them have had to learn 
to deliberately slow down and pace themselves, after they got sick and 
nearly died from simply trying to do to much.)

For those who wish to try it, here's mine:

LDFLAGS="-Wl,-z,now,--as-needed,-O1,--hash-style=gnu,--sort-common"

The "now" bit people will likely want to avoid.  The others should be 
useful, tho there has been discussion about making them Gentoo default, 
and part of them may be default by now (they weren't when I added them, 
tho).

There are some others I've seen as well, but I don't know enough about 
them to use them.  For the ones above (covered in the ld manpage except 
for -Wl, in the gcc manpage, see below):

-Wl is actually a gcc option, telling it to pass what follows to the 
linker.  See the gcc manpage for this one.

-z,now (as opposed to -z,lazy) tells the linker to force all links to be 
resolved at initial program load.  This takes a bit more time /at/ 
initial load, but when an application would otherwise pause for 
additional library functions to load, it now doesn't have to.  It's thus 
a trade-off.  It also has a couple other effects.  Memory use will be 
slightly higher tho not too much, so I'd say it's probably best not to do 
unless you've 2 gigs RAM or more.  But the two effects I use it for are: 
(1) If a library or function from a library can't load, I'd prefer to see 
the error at startup, not later, when the program tries to load that 
bit.  This ensures that.  (2) There are certain somewhat narrow security 
benefits.  I won't attempt to explain them here and they are rather 
narrow, but given I have 8 gigs RAM so that's not an issue, and I 
considered the other aspects slightly positive, with no visible downside, 
I thought it worth it.

I've had this enabled for years, now, with no issues /except/ with the 
ati/radeon drivers when xorg split into modular packages.  I had to turn 
off -z,now for the xf86-ati-radeon package (using the /etc/portage/env/ 
method to do it for just that package) for awhile as a result.  They 
eventually patched the package to filter -z,now, and I'm not sure if it 
has been resolved upstream by now or not, but with the package patched to 
filter it, I don't have to worry about it now.

--as-needed tells the linker to only tag libraries it actually uses as 
NEEDED, not everything on the include line, or indirect dependencies like 
the libraries other libraries it is using use.  The result is that 
unnecessary linkages are avoided, and FAR fewer rebuilds need done when a 
library changes as a result.

I've had this enabled for quite some time now without many issues, 
probably as a result of flame-eyes' testing.  This one should therefore 
be pretty trouble-free now, and has a HUGE upside, but it does still have 
the potential to cause occasional issues on new or obscure packages.  But 
I really DO think the upside is worth it!

-O1 is much like the gcc -O optimization functions, but there's only the 
single -O1 level, for linking.  This one's newer, but I've had no 
problems with it at all, probably because it's commonly enabled and thus 
well tested.

Gentoo's former default for hash was --hash-style=both, but hash-
style=gnu along with -O1 and sort-common were proposed as new defaults.  
I'm not sure if they made it or not, but in any case, this one shouldn't 
be an issue on Linux at all, and I've certainly not had issues at all.  
The other alternative and linker default (before Gentoo's default kicks 
in) is hash-style=sysv.

--sort-common sorts symbols by their alignment size, >=16,8,4,2,1 bytes.  
This packs them tighter than they'd be be otherwise, due to alignment 
constraint issues, thus making for slightly smaller binaries.

I think the --as-needed will make the most difference, and I'm SURE it 
has saved me a LOT of pain, even with the xcb issue alone.  I absolutely 
believe it's worthwhile, and that the pain saved will MORE than make up 
for any occasional issues one /might/ run into with new or obscure 
individual packages.  Note that -Wl is actually a gcc option, telling it 
to pass what follows to the linker.  Thus, the format for /just/ as-
needed would be:

LDFLAGS="-Wl,--as-needed"

> It has to be admitted that these upstream developers are not making life
> any easier for the distribution maintainers.

> But that's the nature of progress, I suppose.  Fortunately, Gentoo can
> give the serious user a set of tools to better deal with these
> inevitable changes.

Agreed.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman