From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id A623E139335 for ; Mon, 28 Jun 2021 06:02:18 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id C0FD6E08CE; Mon, 28 Jun 2021 06:02:17 +0000 (UTC) Received: from mail-ua1-x92d.google.com (mail-ua1-x92d.google.com [IPv6:2607:f8b0:4864:20::92d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 75200E08CE for ; Mon, 28 Jun 2021 06:02:15 +0000 (UTC) Received: by mail-ua1-x92d.google.com with SMTP id o3so6539224uaw.5 for ; Sun, 27 Jun 2021 23:02:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=fOPMr78SvpkIuRTVjaLduDLtGFNYXj47ge8O3xWM4bU=; b=jQ8FpX8c2BcvH//SaQAPXbgkHfnRzn5FRj1UI7Ra2M8RLjthI9hnx57yUiMWu+VxOY yh/a4DSvwPP60XIYAtwwq5V9gWovsEbSY4dPPvUE1R8FXdT56OwrZdUkdffcVtdx+3DD xflig1QyI5O+9+mddM3YehpXOphQsawj3DMAKFuzKGrvgkkF21bSGlMsRUGiyAfgHYQs PWIzedzw+RcHgyhdamNNqHeZiTV4QOKlGbTelDJwk00PkjjhsdoqLl8jz9OTNMK5TQFW mUojwI32ofN2rTyth+2o81etb4YtdhBjV98NQkQfyOY57b/PrxTSSV1RjntYGXNqCoC1 7UHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=fOPMr78SvpkIuRTVjaLduDLtGFNYXj47ge8O3xWM4bU=; b=sOLOada0afrbGBBcLCNh7Y5IFpPGaCMN9tcl4o/G6bd+NEBaIC0aK75E89vQOR0kkL wN0Nw9yxzF0eGc+k52uBhfFVaykue5kGW0HjksU4k3YCt+gl1MqZhuo86LxtNMM6kdpu odgTAEk51dfc1OGrRGzpNqN/ix8H+aHJVBqeRzSTlz7Y9EUSGNGeZH409ORfKyQ+Wv2H DCVTHrh/YA/7v6mY6H6TUIZx4nHiKrtdxnUorcakuXYHd383Ls9rNigowm6oADye4Ypr Y0iH4Kaof+vmHr9Aod0u0ZQGZdWbK9WPjgj5sqPtj9loV1HoSo/udYpY31kLdASUBgm/ iymQ== X-Gm-Message-State: AOAM533BUEtYjXxGro3CIzwUZJLpZQWelDOnmy152kjkqXnp+OUs+3qL hte0y4D9wdd1YLf8wJdOLFnyLf/QwYPxI6PUppD31AhyEtq3uQ== X-Google-Smtp-Source: ABdhPJyjbrimitpJBg68HhLJx8noAl/bLFXNWOH6YEiZZ5in4/j/w6ISi1eBgx0inuK+w4dKpPIZY4iaVlVj13fQBj8= X-Received: by 2002:ab0:140e:: with SMTP id b14mr19383815uae.94.1624860130331; Sun, 27 Jun 2021 23:02:10 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-soc@lists.gentoo.org Reply-to: gentoo-soc@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 From: "Yuan Liao (Leo)" Date: Sun, 27 Jun 2021 23:01:34 -0700 Message-ID: Subject: [gentoo-soc] Week 3 Report for Big Data Infrastructure and H2O ebuilds Project To: gentoo-soc@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" X-Archives-Salt: 1faf8e19-f451-4021-ba77-d7082297d546 X-Archives-Hash: 3035e50a1901463787bdcce6ff6106cb Hi folks, There have been plenty of events going on in my project this week and several pieces of good news. To start with, I placed the last piece of jigsaw of my plans on the two existing Kotlin ebuilds in the Spark overlay, dev-java/kotlin-common-bin and dev-lang/kotlin-bin, by writing up the src_test phase function for each of them that would compile some sample Kotlin programs provided by JetBrains using the Kotlin libraries and the compiler. Then, thanks to fordfrog's assistance, I further improved java-ebuilder and implemented a more precise and reliable way of extracting metadata values from ebuilds, especially each ebuild's SLOT. Last but not least, today I have just created the ebuild for a revolutionary package of kotlin-stdlib that compiles from source, which, to my knowledge, has not appeared yet in any other GNU/Linux distribution. The plan to use sample programs to test the Kotlin compiler was in my original project proposal, but it was not quite easy to implement because compilation of those programs using kotlinc without any extra arguments to it failed, and there were not any instructions to build those tests from the command-line interface. After all, Kotlin heavily relies on Gradle, and the Kotlin Gradle plugin is the de facto tool for building Kotlin projects, so JetBrains, the upstream, has not paid any attention to command-line compiler support. However, the upstream supports building the sample programs from Gradle, and the source code for the Kotlin Gradle plugin is also available, so I had thought of studying how the sample Kotlin programs would be compiled by Gradle by looking at Gradle and the plugin's internals. It later turned out that source code inspection was not necessary at all after I first tried to run Gradle with the '--debug' option, which made Gradle print out the Kotlin compiler option it used to compile those programs. With such information, I successfully implemented dev-lang/kotlin-bin's src_test function by getting the sample program compilation work. Strictly speaking, it is not necessary to test pre-compiled binary packages, but I have modified those packages to install their files in a Gentoo-compliant way, which differs from the upstream's intended package layout, so running those tests would help me and any future maintainers ensure we are modifying the installation paths right. Meanwhile, I contacted fordfrog to discuss the improvements I had made on java-ebuilder and mentioned my effort to let it recognize all kinds of SLOT expressions that might use any combination of plain string literals, Bash string manipulation syntax, and ebuild functions like ver_cut. He suggested that we could skip writing a parser to evaluate the expressions ourselves, and read the evaluated value from ebuild metadata cache generated by Portage's egencache program instead. This method could get us the precise evaluation result, and the implementation would be easy to maintain in the future. A parser, on the other hand, could only mock Portage's behavior to a limited extent and would require updates when anything among Portage, eclasses, Gentoo PMS, etc. is changed. To implement it, I first studied how to get egencache to create cache for an arbitrary ebuild repository that could be at anywhere in the file system, because egencache would by default refuse to create cache for repos not added to Portage, but sometimes users would need to use such a repo. Then, I created a few prototypes, collected feedback on them from fordfrog, incorporated the suggestions, and tuned java-ebuilder for best performance under general use cases. Now, my fork of java-ebuilder is capable of handling all kinds of SLOT expressions. Without fordfrog's insightful advice on a solution that would both be easy to implement and work correctly, I could not have made this improvement, so thanks fordfrog for your help! In my original plan, I did not think of building any Kotlin component from source because Gradle seemed to be a complicated build system to me, and building the entire Kotlin toolchain from Gradle was quite an obscure process. After discovering the way to easily find the Kotlin compiler options used by Gradle and seeing Benda's question about building Kotlin from source, I brought the idea of creating at least a source-based package for the Kotlin Standard Library back to the table. With the same '--debug' trick, I extracted Kotlin compiler options that were used to build multiple components of Kotlin libraries, including kotlin-stdlib, kotlin-stdlib-jdk{7,8} and kotlin-annotations-jvm. Today, I tried to first make an ebuild to compile kotlin-stdlib from source because it was the component that required the least prerequisites to build, and the good news is the ebuild [1] can use the Kotlin compiler and a pre-built bootstrap copy of the Standard Library to build kotlin-stdlib directly from its source code without any compiler errors! The ebuild is not perfect yet, but at least this has proven that some Kotlin components can still be built without Gradle. This not only complies with Gentoo's spirit of building everything from source whenever possible but is probably also the birth of the first Kotlin package for a GNU/Linux distribution that is built from source. Kotlin packages in other distributions' repositories like Arch Linux's AUR are all binary ones exploiting the pre-built distribution archive from JetBrains, just like dev-lang/kotlin-common-bin. I am looking forward to getting rid of the '-bin' suffix in at least the Kotlin library package's name and witnessing creation of the world's first source-based Kotlin package on Gentoo. I have already written an eclass [2] that will help me create source packages for other Kotlin library components next week. Even if I cannot make it, my current work can probably still act as a basis for any future developers to bring such packages to reality. Best regards, Leo [1]: https://github.com/Leo3418/spark-overlay/blob/master/dev-java/kotlin-stdlib/kotlin-stdlib-1.5.10.ebuild [2]: https://github.com/Leo3418/spark-overlay/blob/master/eclass/kotlin-tasks.eclass