[gentoo-soc] Week 7 Report for Big Data Infrastructure and H2O ebuilds Project

[Yuan Liao (Leo)]

Hi folks,

Another week has just passed.  I have been gradually moving on to the next
part of my project where I would fix the ebuilds in the Spark overlay that
are no longer installable, most of which are caused by changes to ebuilds
in the Gentoo repository since last year's GSoC concluded.  At the same
time, a few additional utilities supporting Kotlin on Gentoo have been
created, including an eselect module for Kotlin [1] and some new Kotlin
eclasses.

Many ebuilds in the Spark overlay depend on packages in ::gentoo, but some
dependencies have been broken due to removal of old slots or even deletion
of packages that were really too old.  In the former case, the dependency
could be quickly fixed by updating the slot, although the dependee package
would require some integration tests with the dependency's new slot.  In
the latter case, I would restore the removed ebuild from the Git history of
::gentoo and add it to the Spark overlay.  This is probably not the best
resolution, but as the number of packages in the Spark overlay may suggest,
Apache Spark is a complicated program whose dependency graph spans hundreds
of libraries, so recreating each ebuild removed from ::gentoo from scratch
might take too much time.  However, the ebuilds were not restored
willy-nilly.  ebuilds that used EAPI 5 were ported to EAPI 7, the latest
EAPI supported by the Java eclasses in ::gentoo.  There was an ebuild whose
upstream had been gone and whose SRC_URI was no longer accessible, and I
managed to find a source JAR containing the package's source code on Maven
Central and updated the ebuild to use it instead.

At this point, the next package that needs to be fixed is dev-java/okio,
which is a package containing Kotlin sources and thus should be built with
a Kotlin compiler.  Although my knowledge on Kotlin version compatibility
is limited, I believe there can be situations where a package must be built
by certain versions of the Kotlin compiler.  This, plus fordfrog's interest
in allowing users to keep multiple Kotlin versions in parallel, motivated
me to think about how to implement support for multiple Kotlin compilers.
On Gentoo, the obvious solution to this problem is to create an eselect
module, but that would only allow coarse-grained system-wide control;
chances are some packages require Kotlin 1.4 whereas others need Kotlin
1.5, thus the Kotlin version must be set independently for each individual
package.  For many other programming languages supported by Gentoo,
including Lua, PHP, Python and Ruby, there are USE_EXPAND variables that
can be used for this purpose, but such an approach would not work for
Kotlin yet because USE_EXPAND is defined in Portage profiles, which I have
no power to influence.  As a workaround, I created a new
kotlin-utils.eclass, added a variable that would allow ebuild developers to
define either a single Kotlin version or an open interval of versions
required by the package, and declared another variable for the order of
preference on the accepted versions.  The eclass can search for all Kotlin
versions installed on the system and pick the best one according to the
preference which satisfies any hard requirement.  This would still
compromise the end users' ability to select the Kotlin version used to
build a package, but at least it could allow the ebuild developers to make
a sensible choice for the users.

Support for selecting a Kotlin version from ebuilds is provided by
kotlin-utils.eclass, and the functionality to choose a Kotlin compiler for
the user-space is shipped with the Kotlin eselect module.  The module has a
superset of features offered by the java-vm eselect module.  It implements
not only the notion of user and system compiler but also compiler
preference settings for each Kotlin feature release version.  This feature
is provisioned for any alternative Kotlin compiler packages in the future,
like dev-lang/kotlin with the "-bin" suffix dropped.  For instance, if
users have installed both dev-lang/kotlin:1.5 and dev-lang/kotlin-bin:1.5,
then by using the Kotlin eselect module, they can control whether they want
to use dev-lang/kotlin or dev-lang/kotlin-bin for Kotlin 1.5.  It will be
possible to use different Kotlin compiler packages for different feature
releases too, so users can select dev-lang/kotlin:1.4 and
dev-lang/kotlin-bin:1.5 at the same time.

The version-specific Kotlin compiler preferences are recognized by both
kotlin-utils.eclass and a set of versioned executables for Kotlin tools.
Similar to commands like lua5.4, python3.9 and ruby26, versioned Kotlin
commands like kotlin1.4 and kotlinc1.5 are installed with the Kotlin
compiler packages and supported by the Kotlin eselect module.  With these
versioned commands, users can quickly switch to another Kotlin version
without even invoking eselect.  The versioned commands are not standard in
the Kotlin compiler Zip archive distributed by the upstream (which is what
other distributions' Kotlin packages usually depend on): they are exclusive
on Gentoo, so I hope they will convince any GNU/Linux users who would like
to install multiple Kotlin versions via the system package manager to
consider using Gentoo.

The new kotlin-utils.eclass, the Kotlin eselect module and some additional
Kotlin eclasses are still being tested.  They are working delightfully as
of now, and I am trying to capture more issues and bugs before announcing
and releasing them to more users.  I sincerely hope that these utilities
can form a robust framework for Kotlin packages and programs on Gentoo.

Best regards,
Leo

[1]: https://github.com/Leo3418/eselect-kotlin