From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id E0241139335 for ; Mon, 28 Jun 2021 06:45:05 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id E7E00E089A; Mon, 28 Jun 2021 06:45:04 +0000 (UTC) Received: from smtp.gentoo.org (dev.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 45DC2E089A for ; Mon, 28 Jun 2021 06:45:04 +0000 (UTC) Received: from [2a0c:b641:69c:e781::4] (port=59094 helo=proton) by muon with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1lxl0t-0003vD-3Q; Mon, 28 Jun 2021 06:44:59 +0000 From: Benda Xu To: "Yuan Liao (Leo)" Cc: gentoo-soc@lists.gentoo.org Subject: Re: [gentoo-soc] Week 3 Report for Big Data Infrastructure and H2O ebuilds Project References: Date: Mon, 28 Jun 2021 14:44:57 +0800 In-Reply-To: (Yuan Liao's message of "Sun, 27 Jun 2021 23:01:34 -0700") Message-ID: <87eecmh4ie.fsf@proton.d.airelinux.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-soc@lists.gentoo.org Reply-to: gentoo-soc@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: text/plain X-Archives-Salt: 01559d9a-de98-4d38-bad1-81df0e18be2a X-Archives-Hash: 1cb4c68fcd40076a1b54373151754dd5 Hi Leo, "Yuan Liao (Leo)" writes: > There have been plenty of events going on in my project this week and > several pieces of good news. To start with, I placed the last piece > of jigsaw of my plans on the two existing Kotlin ebuilds in the Spark > overlay, dev-java/kotlin-common-bin and dev-lang/kotlin-bin, by > writing up the src_test phase function for each of them that would > compile some sample Kotlin programs provided by JetBrains using the > Kotlin libraries and the compiler. Then, thanks to fordfrog's > assistance, I further improved java-ebuilder and implemented a more > precise and reliable way of extracting metadata values from ebuilds, > especially each ebuild's SLOT. Last but not least, today I have just > created the ebuild for a revolutionary package of kotlin-stdlib that > compiles from source, which, to my knowledge, has not appeared yet in > any other GNU/Linux distribution. Superb! As far as I know, you are right to be the first one who brings kotlin compiler built from source reproducibly to GNU/Linux community! > The plan to use sample programs to test the Kotlin compiler was in my > original project proposal, but it was not quite easy to implement > because compilation of those programs using kotlinc without any extra > arguments to it failed, and there were not any instructions to build > those tests from the command-line interface. After all, Kotlin > heavily relies on Gradle, and the Kotlin Gradle plugin is the de facto > tool for building Kotlin projects, so JetBrains, the upstream, has not > paid any attention to command-line compiler support. However, the > upstream supports building the sample programs from Gradle, and the > source code for the Kotlin Gradle plugin is also available, so I had > thought of studying how the sample Kotlin programs would be compiled > by Gradle by looking at Gradle and the plugin's internals. It later > turned out that source code inspection was not necessary at all after > I first tried to run Gradle with the '--debug' option, which made > Gradle print out the Kotlin compiler option it used to compile those > programs. With such information, I successfully implemented > dev-lang/kotlin-bin's src_test function by getting the sample program > compilation work. Strictly speaking, it is not necessary to test > pre-compiled binary packages, but I have modified those packages to > install their files in a Gentoo-compliant way, which differs from the > upstream's intended package layout, so running those tests would help > me and any future maintainers ensure we are modifying the installation > paths right. Clever move! Kudos. > Meanwhile, I contacted fordfrog to discuss the improvements I had made > on java-ebuilder and mentioned my effort to let it recognize all kinds > of SLOT expressions that might use any combination of plain string > literals, Bash string manipulation syntax, and ebuild functions like > ver_cut. He suggested that we could skip writing a parser to evaluate > the expressions ourselves, and read the evaluated value from ebuild > metadata cache generated by Portage's egencache program instead. This > method could get us the precise evaluation result, and the > implementation would be easy to maintain in the future. A parser, on > the other hand, could only mock Portage's behavior to a limited extent > and would require updates when anything among Portage, eclasses, > Gentoo PMS, etc. is changed. To implement it, I first studied how to > get egencache to create cache for an arbitrary ebuild repository that > could be at anywhere in the file system, because egencache would by > default refuse to create cache for repos not added to Portage, but > sometimes users would need to use such a repo. Then, I created a few > prototypes, collected feedback on them from fordfrog, incorporated the > suggestions, and tuned java-ebuilder for best performance under > general use cases. Now, my fork of java-ebuilder is capable of > handling all kinds of SLOT expressions. Without fordfrog's insightful > advice on a solution that would both be easy to implement and work > correctly, I could not have made this improvement, so thanks fordfrog > for your help! Great. Thanks fordfrog! > In my original plan, I did not think of building any Kotlin component > from source because Gradle seemed to be a complicated build system to > me, and building the entire Kotlin toolchain from Gradle was quite an > obscure process. After discovering the way to easily find the Kotlin > compiler options used by Gradle and seeing Benda's question about > building Kotlin from source, I brought the idea of creating at least a > source-based package for the Kotlin Standard Library back to the > table. With the same '--debug' trick, I extracted Kotlin compiler > options that were used to build multiple components of Kotlin > libraries, including kotlin-stdlib, kotlin-stdlib-jdk{7,8} and > kotlin-annotations-jvm. Today, I tried to first make an ebuild to > compile kotlin-stdlib from source because it was the component that > required the least prerequisites to build, and the good news is the > ebuild [1] can use the Kotlin compiler and a pre-built bootstrap copy > of the Standard Library to build kotlin-stdlib directly from its > source code without any compiler errors! The ebuild is not perfect > yet, but at least this has proven that some Kotlin components can > still be built without Gradle. This not only complies with Gentoo's > spirit of building everything from source whenever possible but is > probably also the birth of the first Kotlin package for a GNU/Linux > distribution that is built from source. Kotlin packages in other > distributions' repositories like Arch Linux's AUR are all binary ones > exploiting the pre-built distribution archive from JetBrains, just > like dev-lang/kotlin-common-bin. And Debian[1], "the ultimate goal is to build Kotlin from itself." Meaning it is yet to be reality in Debian. > I am looking forward to getting rid of the '-bin' suffix in at least > the Kotlin library package's name and witnessing creation of the > world's first source-based Kotlin package on Gentoo. I have already > written an eclass [2] that will help me create source packages for > other Kotlin library components next week. Even if I cannot make it, > my current work can probably still act as a basis for any future > developers to bring such packages to reality. Best, Benda > [1]: https://github.com/Leo3418/spark-overlay/blob/master/dev-java/kotlin-stdlib/kotlin-stdlib-1.5.10.ebuild > [2]: https://github.com/Leo3418/spark-overlay/blob/master/eclass/kotlin-tasks.eclass 1. https://wiki.debian.org/Kotlin