Re: [gentoo-user] Apache died this morning... why?

[Alan McKinnon]

Replies inter-posted


On 30/12/2013 14:25, Tanstaafl wrote:
> On 2013-12-30 6:30 AM, Alan McKinnon <alan.mckinnon@×××××.com> wrote:
>> To see what's going on, run ldd on:
>>
>> /usr/lib64/apache2/modules/libphp5.s
>
> Result:
>
>>  # ldd /usr/lib64/apache2/modules/libphp5.so
>> ldd: warning: you do not have execution permission for
>> `/usr/lib64/apache2/modules/libphp5.so'
>>         linux-vdso.so.1 (0x00007fffc3cbf000)
>>         libc-client.so.1 => /usr//lib64/libc-client.so.1
>> (0x00007f279599d000)
>>         libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f279577b000)
>>         libreadline.so.6 => /lib64/libreadline.so.6 (0x00007f2795535000)
>>         libaspell.so.15 => /usr//lib64/libaspell.so.15
>> (0x00007f2795263000)
>>         libm.so.6 => /lib64/libm.so.6 (0x00007f2794f69000)
>>         libssl.so.1.0.0 => /usr//lib64/libssl.so.1.0.0
>> (0x00007f2794cff000)
>>         libcrypto.so.1.0.0 => /usr//lib64/libcrypto.so.1.0.0
>> (0x00007f279491a000)
>>         libz.so.1 => /lib64/libz.so.1 (0x00007f2794703000)
>>         libmcrypt.so.4 => /usr//lib64/libmcrypt.so.4 (0x00007f27944d1000)
>>         libdl.so.2 => /lib64/libdl.so.2 (0x00007f27942cd000)
>>         libonig.so.2 => /usr//lib64/libonig.so.2 (0x00007f2794062000)
>>         libt1.so.5 => /usr//lib64/libt1.so.5 (0x00007f2793e03000)
>>         libfreetype.so.6 => /usr//lib64/libfreetype.so.6
>> (0x00007f2793b64000)
>>         libpng15.so.15 => /usr//lib64/libpng15.so.15 (0x00007f2793939000)
>>         libjpeg.so.8 => /usr//lib64/libjpeg.so.8 (0x00007f27936e4000)
>>         libdb-4.8.so => /usr//lib64/libdb-4.8.so (0x00007f279336a000)
>>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f279314c000)
>>         libgdbm.so.3 => /usr//lib64/libgdbm.so.3 (0x00007f2792f46000)
>>         libcurl.so.4 => /usr//lib64/libcurl.so.4 (0x00007f2792ceb000)
>>         libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f2792ada000)
>>         libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f2792873000)
>>         libxml2.so.2 => /usr//lib64/libxml2.so.2 (0x00007f2792512000)
>>         libmysqlclient.so.16 => /usr/lib64/mysql/libmysqlclient.so.16
>> (0x00007f279218b000)
>>         libnetsnmp.so.30 => /usr//lib64/libnetsnmp.so.30
>> (0x00007f2791eb0000)
>>         libc.so.6 => /lib64/libc.so.6 (0x00007f2791b0a000)
>>         libpam.so.0 => /lib64/libpam.so.0 (0x00007f27918fb000)
>>         libncurses.so.5 => /lib64/libncurses.so.5 (0x00007f27916d8000)
>>         libstdc++.so.6 =>
>> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libstdc++.so.6
>> (0x00007f27913d3000)
>>         /lib64/ld-linux-x86-64.so.2 (0x00007f279681b000)
>>         librt.so.1 => /lib64/librt.so.1 (0x00007f27911ca000)
>>         libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f2790f95000)
>>         libgcc_s.so.1 =>
>> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libgcc_s.so.1 (0x00007f2790d7e000)


looks fine


>
> and
>
>> /usr//lib64/libcurl.so.4
>
>>  # ldd /usr//lib64/libcurl.so.4
>>         linux-vdso.so.1 (0x00007fffa7bff000)
>>         libssl.so.1.0.0 => /usr/lib64/libssl.so.1.0.0
>> (0x00007f510232b000)
>>         libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0
>> (0x00007f5101f46000)
>>         libz.so.1 => /lib64/libz.so.1 (0x00007f5101d2f000)
>>         librt.so.1 => /lib64/librt.so.1 (0x00007f5101b27000)
>>         libc.so.6 => /lib64/libc.so.6 (0x00007f5101781000)
>>         libdl.so.2 => /lib64/libdl.so.2 (0x00007f510157c000)
>>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f510135f000)
>>         /lib64/ld-linux-x86-64.so.2 (0x00007f51027fb000)


looks fine


>
> Doesn't mean anything to me though... ;)

It's just a list of the libs a file knows it is linked to.
First is the lib name then the big arrow (=>) then the file containing
that lib then a bunch of numbers. Ignore the numbers, pay most attention
to anything that says "not found" - that's the junk revdep-rebuild looks for


>
>> preserved-rebuild should just take care of all this automagically.
>> Do you have preserve-libs in FEATURES?
>
> Nope... is this now recommended? Is it the default on new installs?

Yes it's the default for new installs and comes highly recommended
(unless you like having stuff not work at all till revdep-rebuild
completes...)

There was a news item 2013-06-07:

2013-06-07-portage-preserve-libs-default
  Title                     Portage preserve-libs default
  Author                    Zac Medico <zmedico@g.o>
  Posted                    2013-06-07
  Revision                  1

Beginning with sys-apps/portage-2.1.12, FEATURES=preserve-libs is
enabled by default. Even though preserve-libs makes it unnecessary to
use revdep-rebuild for most common updates, it is still a good practice
to run `revdep-rebuild -ip` after updates, in order to check if there
are any broken library dependencies that preserve-libs was not able to
handle. For example, see http://bugs.gentoo.org/show_bug.cgi?id=459038.

If you would like to disable preserve-libs by default, then set
FEATURES="-preserve-libs" in make.conf. See the make.conf(5) man page
or the following wiki page for more information:

http://wiki.gentoo.org/wiki/Preserve-libs

>
>> Do a pretend run of revdep-rebuild. I'll bet you end up rebuilding curl
>> and/or php, but not apache.
>
> Actually, I did that right after the updates and it didn't recommend
> anything (I always do revdep-rebuild -p after any system updates like
> gcc, glib/c, etc)...
>
>> Apache is unlikely to be at fault, it loads a dynamic module and use it,
>> that module either works or it doesn't.
>
> Ok... so, the question is still why did it die?
>
> This happened by the way when the logs were rotated by logrotate. Maybe
> that is significant?

Yes, that is highly significant.

IIRC logrotate can work in one of two ways:

1. rename the log file and create a new empty one
2. copy the log file elsewhere and truncate the original

I forget which way it does it for the moment...

#1 is fast but leaves the daemon (apache or syslog) trying to write to a
file that isn't there anymore. Or worse, it's writing to an open file
that has been deleted and a new one with the same name still exists.
#2 is slower but safer.

Either way, the apache daemon has to be told it's log file went away.
Not all daemons can use inotify to just find this out, some have to be
told, so logrotate resets/restarts/hups them. In the case of apache it
does a graceful restart (what you get with apachectl graceful).

Your apache re-read it's config file at that point, found any error for
php and decided to roll over and die. It's designed to do that (nagios
os similar btw would have told you this happened)



>
>>> The GCC Upgrade guide is a bit outdated (still referring to gcc 3.4 and
>>> 4.1, with no mention of newer versions, so I wasn't sure if that was
>>> still necessary...
>
>> According to the posted error, this has nothing to do with compiler
>> versions, it is linker errors related to glibc
>>
>> You do not have to rebuild system, world or the known universe. You only
>> have to do that when the a gcc upgrade changes the data format on-disk
>> that the C++ compiler generates. That has not happened here.
>>
>> There's an insane amounts of FUD around about rebuilding gcc, all of it
>> originating from ricers without a clue. You run strictly stable-only so
>> never fear, if a gcc upgrade required a world rebuild you would have
>> already been subjected to 12-month long threads about it right here on
>> this list
>
> I know, and the GCC upgrade guide is pretty clear on that point, and
> since I didn't say anything about rebuilding anything other than
> sys-devel/libtool, which it does specifically mention, I'm not sure why
> you brought that up in this context.
>
> I just did it (rebuilt libtool) anyway, maybe I should do  the same for
> curl and/or php...

It won't do any harm. php might take a while to rebuild though... ;-)


>
> No worries, apache is only used for local admin stuff anyway, so it
> isn't like anyone other than me will notice if it dies... :)



-- 
Alan McKinnon
alan.mckinnon@×××××.com