Re: [gentoo-server] SSH, updates and serial port switches

[Justin Filoseta]

On Friday 12 November 2004 10:47 am, Richard Yumul wrote:

> > I have seen in different FAQs that running a serial cable to each
> > server and using a SSH serial console switch is a good idea, but I
> > am having trouble finding something cheap in this arena. Also, how
> > much better is it in terms of reliability in case something goes
> > really wrong with the server? FYI, all the servers are plugged into
> > a remote APC reboot switch but I almost never use this, as many
> > times it ends up invalidating the filesystem and therefore
> > requiring a physical intervention at the keyboard. Anyway around
> > this problem as well?

> This actually happened to me w/ a recent emerge;  What saved me is I
> had a KVM over IP devices.  It was a life saver...
>

While KVM over IP is handy, I have always seen it is as a hack compared 
to a good serial console, although that is probably because I started 
with Sun boxes in the server room and they are just designed to be used 
over serial console. Either way, to me it seems foolish to run a 
production server with the only remote access being served by ssh. I 
don't mean to imply anyone here is a fool, just that the times I have 
needed a secondary way in, I was very thankful one was available.

Before I get into the serial console discussion, having only used KVM 
over IP a few times I am curious, is it possible to send emergency 
SysReq key sequences through them?

A little background for the original poster:
With a proper serial console setup one can access and control the bios 
(depending on the class of board), the grub console menu, and the 
entire kernel boot process including login. That includes the ability 
to interact with the system if it gets caught waiting for manual 
intervention as well. 

At work I use an external serial console server designed for the task, 
if you go this route, I would recommend one with RJ45 connectors, cat5 
is much easier to deal with and you just put an adapter on the serial 
port of every box. That way when racks are wired up it is just another 
network cable running to each box. Perle CS9000 series have served me 
well, but I am sure there are better boxes out there.

Since cost was mentioned as a concern, valid considering serial console 
servers can easily top $1k, remember that there is nothing special 
about a serial console. Any *nix box can serve as a serial console for 
another machine, the two boxes just need to be connected over a serial 
port. I have seen this take the form of one old machine that was 
stuffed full of serial ports and used in lieu of purchasing a serial 
console. However, I have also seen this ability used to form a web 
between servers. It was a little tricker and required documentation 
indicating that serverA was the console for serverB and serverB was the 
console for serverC which in turn was serving for serverA (simplified a 
bit), but it was a usable serial console setup for the cost of a few 
cables.

As with any second way in, you need to consider the security 
implications, since compromise of a serial console server provides tty 
access to at least one other machine, although normally sitting at the 
login promt. A properly secured serial console has always been worth 
the risk for me, but your environment may be different.

Be it serial console or KVM over IP, the ability to drop to single user 
or even boot init=/bin/sh to perform those serious operations is 
invaluable. Plus, if you plan ahead and attach a backup drive (usb pen, 
cdrom left in the tray, second HD ... anything), when the system gets 
totally hosed it can be booted off the backup medium and fixed or 
re-installed without ever having to set foot in the server room. That 
is of course if it passes the BIOS post ;)

Justin