[gentoo-dev] Maintaining production systems - and losing ebuilds

[Ron OHara]

Hi,

I want to raise an issue resulting from my experience so far in using 
Gentoo as the basis of production systems. Some may ask why? - but 
basically 'portage' seems to offer the very best framework for ongoing 
maintenance/admin of systems, though it's not perfect in that role.

In essence, the continuous, easy upgrade capability of portage is great 
for a development system and should be an excellent mechanism for 
critical security (and other) upgrades in a production environment (and 
it is).
The problems arise because of the continuous easy upgrades!! - the main 
benefit is also the main problem.

I have just hit a real life hassle with a security upgrade. The history 
of it goes like this:

[background]
The example system in trouble is an old P233, and used to be on the end 
of a dialup link (it's now ADSL).
Gentoo has been installed for about 10 months and the last time it was 
brought completely up to date was about 6 months ago (emerge rsync && 
emerge -u world)
[/background]


[creating a problem]

As you have guessed, I've just had some system problems - partly of my 
own creation, but partly because of how Gentoo operates.

My real problem came from doing 'emerge rsync', and then just 
(selectively) doing 'emerge -u openssl'

This installed 'openssl-0.9.7' and removed 'openssl-0.9.6' - 
unfortunately lots of stuff on the system was compiled and linked 
against 'openssl-0.9.6' and they promptly broke. IE. Serious outage on a 
production system.

There is a script designed to fix this called 'revdep-rebuild' which 
scans all the installed binaries for broken dependencies and then 
recompiles them which should make them link against the nice new 
'openssl-0.9.7'

except!!! - revdep-rebuild carefully tries to recompile the exact 
versions of software you have installed (good idea) - but the Gentoo 
central repository has since deleted some of the build scripts for these 
older versions and when I did the 'emerge rsync', the scripts were also 
removed from my system. So I ended up where I am now - I have to go 
through and do 'emerge -u world' and then 'revdep-rebuild' to get it all 
working... not nice when there are nearly 200 packages to 
download/recompile on an old P233

[/creating a problem]




As you can see, I was intending to leave the installed set of packages 
(and versions) alone. For this machine (and any production system), I 
dont want to install each and every little patch as it comes along. The 
machine is 'stable' - so I only want to apply upgrades on a very 
selective, controlled, manual basis - but still use portage for the 
package management.
This is a very common tactic for 'production' machines, where you want 
the minimum number of changes to reduce your risks of outage.

The trap is that 'emerge rsync' removes old .ebuilds that your installed 
machine may need if revdep-rebuild is to be able to recovery things 
after a critical library is rebuilt.
In the way portage works, the only time it is safe for 'emerge rsync' to 
delete ebuilds, is immediately after successfully doing 'emerge -u world'.


Is there a way to suppress the 'delete' part of  rsync? Maybe a setting 
in /etc/make.conf ?

That way, even though Gentoo may have removed the relevant (old) ebuild 
I want, the target machine would have it's local portage version for 
future recompiles.... I can afford the disk space!!!




Regards
Ron OHara
PS: This is not a 'casual' problem for me - I've convinced a client to 
use Gentoo for the basis of their deployments and the plan is supposed 
to be for around 900 sites!! - catering for production software support 
for the next decade is very relevant to things in this scenario.








--
gentoo-dev@g.o mailing list