[gentoo-portage-dev] Multiple language use, `component-based' design, and other issues

[Jeremy Maitin-Shepard]

It has been mentioned that portage-ng will be written using multiple
programming languages.  I see a number of compelling reasons to avoid
using multiple languages, while I see no particular compelling advantage.

First, supporting multiple languages will mean that significant time and
energy will need to be spent developing and maintaining the
inter-language interfaces.  The result will be a system that is less
efficient (inter-language interfaces tend to be less efficient then
intra-language ones) and more difficult to maintain and extend (because
extending will mean that these inter-language interfaces will need to be
extended correspondingly).  Furthermore, inter-language interfaces tend
to be less elegant syntactically than intra-language ones, meaning that
the resulting code will generally be less elegant (and thus more
difficult to follow).

The Python-Bash inter-operation in the current portage is a clear
example of the inefficiency involved in language interoperability
(certainly, most will not be as inefficient as Bash).  This problem can
be avoided in portage-ng in a number of ways, which include stricter
specification of how meta-data variables can be specified in the ebuild
files, separating ebuilds into multiple files, and developing a new
bash-like language specific to portage and writing a parser and
interpreter for this language which is included in portage.  (In order
for the development of portage-ng to begin, it would be useful to
decide on issues like this.)

A second disadvantage to multiple language use is that it will increase
the compile-time dependencies (and depending on the language, possibly
the runtime dependencies) of portage unnecessarily.  If portage is
written using, for example, Python, Prolog, and Ruby, then _EVERY_
Gentoo system that compiles portage will need Python, Prolog, and Ruby
installed, and possibly have them to run portage also.  Python is less
of a problem because it is relatively common, but Prolog and Ruby are
often not found on systems.  Additional dependencies further complicate
the handling in portage of those dependency packages (as is the case
currently with python), especially if they are a runtime dependency.
Additional dependencies also bloat the size of the stage tarballs.

A third issue is that while we all have our pet language, and in
principle it might seem useful to support `all' languages, so that we
can each use our language of choice in writing portage, in practice
support for each language will likely require significant effort to
write the necessary interface code.  Furthermore, and perhaps a greater
problem, in practice it will be necessary for portage developers to
learn all of the languages used in the various parts of portage.  Thus,
instead of some people having to learn and use a single language that
they might prefer to avoid, all developers would need to learn and use
multiple languages which they would prefer to avoid.

The main advantage cited for using multiple languages has been that
certain languages are `better' for doing certain types of things.  For
example, it has been argued that Prolog should be used for dependency
calculation because programs written in it can be proven correct.  I
fail to understand why it is particularly important that the portage
dependency checker be ``provably correct.''  To my knowledge, there have
been no significant problems with there being bugs in the current
dependency checker.  It is also useful to note that there are tens of
thousands or more software programs, far more critical than a portage
dependency checker, which are not written in languages in which they can
be proven correct, but which operate quite adequately.  It is perhaps
useful for the software on the Spirit Mars rover to be provably correct,
but that is simply not a useful guideline for the portage dependency
checker.  It has also been argued that it is more convenient to write a
dependency checker in Prolog, compared to other languages.  I can
assure the reader that implementing a topological sort algorithm in any
language is not overly complex; certainly not sufficiently complex to
justify doing it in another language and dealing with language
interoperability problems and adding an additional compile-time and
possibly runtime dependency.

It has been mentioned that portage-ng will have a `component-based'
design.  It is not clear what the term `component-based' means exactly,
and so it would be useful to get some clarification.  To me that term
suggests that portage-ng will use some sort of complex runtime dynamic
linking model.  I would argue that it may be simpler to simply handle
all optional functionality, if there is any, through the use of USE
flags, rather than going to the trouble of supporting such a complex
model.  Clearly, some type of static linking could only be supported if
portage-ng did not depend on a runtime dynamic linking model.
Advantages to allow static linking include greater robustness in cases
of failure of various shared libraries, and the dependencies of portage
will not need to be handled as carefully, because there will be no risk
of breaking a statically-linked portage.

-- 
Jeremy Maitin-Shepard