[gentoo-dev] Fixing the TERM mess

[Ciaran McCreesh]

[ Please excuse the length... I'm making a proposal related to an issue
which isn't widely understood, and I think it's better to provide an
explanation straight off rather than watch this thread descend into
irrelevant arguments by people who don't have a clue about what's being
discussed. ]

First, some background. Different terminal emulators have different
capabilities.  Some can report their width and height to applications,
some can't. Some can do 8 colours, some can do 8 colours + bold, some
can do 16 colours, some can do 16 colours + bold, some can do 88 or 256
colours + bold and italic. Some terminals have problems writing to the
last character on the screen, some have strange erase bugs, some allow
programs to resize the terminal. Different terminals use different magic
sequences to do these things. Keyboard handling is also different
between different terminals -- some terminals don't support the meta /
alt key, some don't distinguish between escape and control, and handling
of keys like home, end and the function and arrow keys varies
considerably.

All modern terminals will set the TERM environment variable. This allows
applications to:

* support optional capabilities (eg colour) on terminals where it is
  available.

* work around bugs or limiting features in terminals, for example by
  never writing to the bottom right character on terminals that don't
  support it properly.

* know which escape sequences to use when reading or writing from the
  terminal.

Obviously, making each application have its own list of TERM values and
settings isn't a very good solution -- it's a lot of duplicated
information, and makes it hard to update things when new terminals are
released. A centralised database and library makes much more sense.

Historically, there have been two widely used implementations of said
database. These are usually known as 'termcap' and 'terminfo'.

The termcap method is provided by libtermcap-compat. Most applications
which use this method only do so as an option for systems where terminfo
is not available -- for example, for Vim, terminfo vs termcap is a
compile-time option. The termcap database is limited, generally out of
date and mostly unmaintained. It will likely not remain in the tree for
much longer (bug #103105).

The terminfo method is provided by ncurses. It is considerably more
powerful and has far fewer arbitrary restrictions upon database entry
lengths. It is also less unmaintained than termcap. In simple terms,
ncurses provides two things -- a standardised way for applications to
look up terminal capabilities, and a set of library functions which
simplify various terminal-related tasks. Some applications use both of
these, whilst others only use the former and implement their own more
powerful screen handling. Both approaches are valid, depending upon
application needs.

Now, there's a slight problem. If you have TERM=shinynewterm, and you
ssh to a box with an old terminfo database, you'll get a warning or
error that your terminal isn't recognised when you try to use an
ncurses-based application. You can either ask the sysadmin to upgrade,
or you can install the relevant terminfo files into ~/.terminfo. This
isn't a major problem, but the local terminfo thing isn't very well
known, so some people have this silly misconception that you're either
screwed or have to export a fake TERM if the box you're on doesn't
recognise your terminal.

Unfortunately, some of the people who have this silly misconception also
happen to write terminal emulators.

See, certain terminal emulators lie about their TERM setting. Usually
it's things that aren't xterm pretending to be an xterm, although rxvt
sometimes crops up too. Examples of things pretending to be xterm
include Konsole, Gnome Terminal.

The logic behind it goes like this:

* We don't understand this 'terminfo' thing, but we want our terminal to
  work with programs which use ncurses.

* We've implemented xterm-style colour sequences and some basic cursor
  movement.

* Therefore, we shall set TERM=xterm.

This is not a good idea. See, real xterm supports a hell of a lot of
fancy voodoo. It and rxvt-unicode are two of the most fully featured
terminal emulators (from a terminal capability point of view) out there.
None of these cheap knockoffs that claim to be xterm come anywhere near
close to supporting full xterm capabilities.

Right now we have two sets of workarounds in place to avoid problems
with these lying apps. The first is that the xterm terminfo entry is
utterly crippled. The second is that apps which make full use of all the
fancy terminal sequences will often include additional checks and
restrictions when running on an 'xterm' (for example, Vim will disable
most of its mouse capabilities on an xterm to avoid breaking Gnome
Terminal).

This is bad.

It screws over xterm users, because their terminal is not being used to
full effect. Applications which should support 256 colours, full mouse
support, background detection, terminal titles etc have to disable
support for those things in case they break fake xterms.

It also screws over users of those things that pretend to be xterm. Both
Konsole and Gnome Terminal implement a few extras beyond the 'crippled'
xterm definition, but these can't be turned on for fear of breaking both
other "pretend to be xterm" terminals and real xterms.

For some apps, this doesn't really make much difference. For others it
does. As it stands, applications such as powerful text editors and
terminal multiplexers (screen) can't be used properly on anything that
claims TERM="xterm".

At this point, there's at least one person who's going to whine "but I
use vim and screen on $whatever just fine!". Which is true, for
sufficiently small values of fine. As soon as you try to do 256 colour
support (`:set t_Co=256 | colorscheme inkpot` in vim, for example), or
anything mouse related, you're stuffed. You'll also be experiencing
considerable screen redraw slowdowns because some wannabe-xterms don't
do screen refresh voodoo properly.

Not satisfied, and still think your crappy Gnome Terminal is a full
xterm? Install xtermcontrol, run `xtermcontrol --get-bg` and watch your
terminal get very very upset. This is a frickin' nuisance, because
there's a long standing issue to do with Vim and making the default
colour scheme readable on both light and dark backgrounds. It would be
easy to fix if we could count upon anything claiming to be an xterm
supporting xterm colour reporting, but this is not the case.

Now the proposal. This isn't something that can happen immediately, but
it's something I'd like to see us working towards:

* Make everything that isn't xterm set its own TERM value. Possibly the
  same for things pretending to be rxvt, although this is less of an
  issue since I think everything that pretends to be rxvt is "rxvt plus
  some patches".

* Install, either with the terminal (as is done by rxvt-unicode
  currently), or as part of ncurses, proper terminfo definitions for
  these terminals.

* De-cripple the standard xterm definition and remove restrictions from
  programs which can make full use of xterm's capabilities.

* Provide a short FAQ explaining what I said above in a more
  newbie-friendly way, along with a description of how to generate and
  use terminfo files under a user's home directory for compatibility
  with legacy systems and a note that users sshing from legacy systems
  to Gentoo may need to switch their TERM value.

* Include TERM stuff in policy so that the problem doesn't crop up again
  a few months later.

* Putty? Since I consider sshing from insecure systems irresponsible, I
  don't know much about this one, other than that it's another dirty
  liar and that said lying has caused various bugs in the past. Perhaps
  someone could clarify with details of what it pretends to support and
  what it really supports?

Before I go any further with this, does anyone have a really really good
reason why we should continue to make lots of users suffer because a few
upstreams don't know what they're doing?

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Shell tools, Fluxbox, Cron)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm