[gentoo-soc] Week 3 Report for Big Data Infrastructure and H2O ebuilds Project

[Yuan Liao (Leo)]

Hi folks,

There have been plenty of events going on in my project this week and
several pieces of good news.  To start with, I placed the last piece
of jigsaw of my plans on the two existing Kotlin ebuilds in the Spark
overlay, dev-java/kotlin-common-bin and dev-lang/kotlin-bin, by
writing up the src_test phase function for each of them that would
compile some sample Kotlin programs provided by JetBrains using the
Kotlin libraries and the compiler.  Then, thanks to fordfrog's
assistance, I further improved java-ebuilder and implemented a more
precise and reliable way of extracting metadata values from ebuilds,
especially each ebuild's SLOT.  Last but not least, today I have just
created the ebuild for a revolutionary package of kotlin-stdlib that
compiles from source, which, to my knowledge, has not appeared yet in
any other GNU/Linux distribution.

The plan to use sample programs to test the Kotlin compiler was in my
original project proposal, but it was not quite easy to implement
because compilation of those programs using kotlinc without any extra
arguments to it failed, and there were not any instructions to build
those tests from the command-line interface.  After all, Kotlin
heavily relies on Gradle, and the Kotlin Gradle plugin is the de facto
tool for building Kotlin projects, so JetBrains, the upstream, has not
paid any attention to command-line compiler support.  However, the
upstream supports building the sample programs from Gradle, and the
source code for the Kotlin Gradle plugin is also available, so I had
thought of studying how the sample Kotlin programs would be compiled
by Gradle by looking at Gradle and the plugin's internals.  It later
turned out that source code inspection was not necessary at all after
I first tried to run Gradle with the '--debug' option, which made
Gradle print out the Kotlin compiler option it used to compile those
programs.  With such information, I successfully implemented
dev-lang/kotlin-bin's src_test function by getting the sample program
compilation work.  Strictly speaking, it is not necessary to test
pre-compiled binary packages, but I have modified those packages to
install their files in a Gentoo-compliant way, which differs from the
upstream's intended package layout, so running those tests would help
me and any future maintainers ensure we are modifying the installation
paths right.

Meanwhile, I contacted fordfrog to discuss the improvements I had made
on java-ebuilder and mentioned my effort to let it recognize all kinds
of SLOT expressions that might use any combination of plain string
literals, Bash string manipulation syntax, and ebuild functions like
ver_cut.  He suggested that we could skip writing a parser to evaluate
the expressions ourselves, and read the evaluated value from ebuild
metadata cache generated by Portage's egencache program instead.  This
method could get us the precise evaluation result, and the
implementation would be easy to maintain in the future.  A parser, on
the other hand, could only mock Portage's behavior to a limited extent
and would require updates when anything among Portage, eclasses,
Gentoo PMS, etc. is changed.  To implement it, I first studied how to
get egencache to create cache for an arbitrary ebuild repository that
could be at anywhere in the file system, because egencache would by
default refuse to create cache for repos not added to Portage, but
sometimes users would need to use such a repo.  Then, I created a few
prototypes, collected feedback on them from fordfrog, incorporated the
suggestions, and tuned java-ebuilder for best performance under
general use cases.  Now, my fork of java-ebuilder is capable of
handling all kinds of SLOT expressions.  Without fordfrog's insightful
advice on a solution that would both be easy to implement and work
correctly, I could not have made this improvement, so thanks fordfrog
for your help!

In my original plan, I did not think of building any Kotlin component
from source because Gradle seemed to be a complicated build system to
me, and building the entire Kotlin toolchain from Gradle was quite an
obscure process.  After discovering the way to easily find the Kotlin
compiler options used by Gradle and seeing Benda's question about
building Kotlin from source, I brought the idea of creating at least a
source-based package for the Kotlin Standard Library back to the
table.  With the same '--debug' trick, I extracted Kotlin compiler
options that were used to build multiple components of Kotlin
libraries, including kotlin-stdlib, kotlin-stdlib-jdk{7,8} and
kotlin-annotations-jvm.  Today, I tried to first make an ebuild to
compile kotlin-stdlib from source because it was the component that
required the least prerequisites to build, and the good news is the
ebuild [1] can use the Kotlin compiler and a pre-built bootstrap copy
of the Standard Library to build kotlin-stdlib directly from its
source code without any compiler errors!  The ebuild is not perfect
yet, but at least this has proven that some Kotlin components can
still be built without Gradle.  This not only complies with Gentoo's
spirit of building everything from source whenever possible but is
probably also the birth of the first Kotlin package for a GNU/Linux
distribution that is built from source.  Kotlin packages in other
distributions' repositories like Arch Linux's AUR are all binary ones
exploiting the pre-built distribution archive from JetBrains, just
like dev-lang/kotlin-common-bin.

I am looking forward to getting rid of the '-bin' suffix in at least
the Kotlin library package's name and witnessing creation of the
world's first source-based Kotlin package on Gentoo.  I have already
written an eclass [2] that will help me create source packages for
other Kotlin library components next week.  Even if I cannot make it,
my current work can probably still act as a basis for any future
developers to bring such packages to reality.

Best regards,
Leo

[1]: https://github.com/Leo3418/spark-overlay/blob/master/dev-java/kotlin-stdlib/kotlin-stdlib-1.5.10.ebuild
[2]: https://github.com/Leo3418/spark-overlay/blob/master/eclass/kotlin-tasks.eclass