From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id F409513933E for ; Mon, 12 Jul 2021 03:56:57 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id E1549E0864; Mon, 12 Jul 2021 03:56:56 +0000 (UTC) Received: from mail-vs1-xe35.google.com (mail-vs1-xe35.google.com [IPv6:2607:f8b0:4864:20::e35]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 7AF64E0864 for ; Mon, 12 Jul 2021 03:56:56 +0000 (UTC) Received: by mail-vs1-xe35.google.com with SMTP id bf5so2151202vsb.4 for ; Sun, 11 Jul 2021 20:56:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=RWZEtNuuQ9Tgtc6m+4DISWCldqBrbF93AH3YrxeOQBw=; b=tDPSsUoiEWYCT6sVbSZVgJlfojG69Oj6HeYCskhmlpLe5AYCgnnY9TEIfNXbJKDqJr ut9obNUJ8traWz94duHUAOS0Bj6edI0Jc/srZJzghGfcD1G6zoIqbehSi4IoACxKVBFY sKLoBwgH7QhFNwPLy+K8/E4wZQDx2xwTTX+lqAonm7Hnb4r3jJqJ3nSx7EZTQsLgA8Mn Pax7lE/gxL1aI1iFQEWYquaurR7bygd6K+FrWdiMgWbIGmQdcDVncW/PxaRZyt7frpJD rgby004jq6UGM5RKgJ8kUCuF+YMNn3AxjAYa6O/4nQVbX/06I/WwjslShPosTS/ddk3q IO8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=RWZEtNuuQ9Tgtc6m+4DISWCldqBrbF93AH3YrxeOQBw=; b=qaEnw7DKmNyCMN0e9qFJcKzAnGYZEcItgUcZMxdusWNhA6r+BO3NvrROUaM+3PXp4J VMxcRzqQF+QooBj8dEEzS9aR6bWm7l9wAv0ZJXcCYLt7KSQesfK6FYyK2yQiyc7CfJ4o FiFhJGAc6Ib1Fn2qVUbErP4hA5kSA0eQkgMiabHR9TlzVR+eP3vpbePIxjBHjjT7PPaN 08/E1BcQyLP+cTaUVd5FqdWIIT456jpd0JK1CdRP+5cncOHEgJXAeXSZZKGVeuCUyTHH 5DpEcDfO/o+1aJ7zGgO8k7xPMMtiKWUZ44S72aBVzggF1Dqh8ucqPjoOOTbwwHMT7srt LM+g== X-Gm-Message-State: AOAM533ymSM9Eqlby/n557VyGoY+favPQDgy6vGGxipluZgXMeRpMcFQ qzLlM7msq+skoPeud39Y6EL9jIeaet8yWnk8bwh2IZFJWIQ= X-Google-Smtp-Source: ABdhPJy8iwK7BXBB94I6XgzE0n7kkZ/DRoRqzpDrjsHy6HLFffaybt+mcB/GTogEUY/zJzbs5sE9NcaVdZ1PZhtuCHk= X-Received: by 2002:a67:c789:: with SMTP id t9mr35088183vsk.60.1626062215413; Sun, 11 Jul 2021 20:56:55 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-soc@lists.gentoo.org Reply-to: gentoo-soc@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 From: "Yuan Liao (Leo)" Date: Sun, 11 Jul 2021 20:56:19 -0700 Message-ID: Subject: [gentoo-soc] Week 5 Report for Big Data Infrastructure and H2O ebuilds Project To: gentoo-soc@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" X-Archives-Salt: 1acd6e0e-f949-474c-9dac-fb956b5e8868 X-Archives-Hash: f7f0be4bf651992ce89cb56f2dfdaafe Hi folks, This week, I have moved on to the next part of my project: an ebuild testing framework. For sophisticated ebuilds with many dependencies and a lot of USE flags, a framework which facilitates automated ebuild installation tests can be very useful. Such a testing framework can also integrate with a CI service so the tests can be run periodically and automatically, reminding developers of ebuild breakages early. Maintainers of ebuilds for Java packages -- especially those ebuilds in the Spark overlay -- might want to test installing those ebuilds periodically and with different USE flag configurations. To start with, Java packages usually have multiple dependencies, and successful installation of all those dependencies is necessary for successful installation of such a Java package itself. Some packages in the Spark overlay depend on ebuilds in ::gentoo, but chances are the latter set of ebuilds get updated or removed, preventing those ebuilds in the overlay from being merged properly. As a matter of fact, some ebuilds in the Spark overlay are already non-installable due to removal of some Java packages from ::gentoo. In this case, if there is a set of ebuild installation tests that run every day or every week, then problems like this can be discovered as soon as they emerge. Besides dependencies, USE flags are another type of parameters that can determine whether an ebuild can be installed because a USE flag can affect the list of dependencies and/or how the ebuild is compiled and installed. Many ebuilds in the Spark overlay have a 'binary' USE flag (just like any ebuild created by java-ebuilder) which installs a pre-built binary JAR from Maven Central when enabled. Some of the ebuilds can be installed with USE="binary" but not with USE="-binary" due to errors in Java source compilation. The same ebuild should be tested with both of the USE flag configurations if the goal is to support both binary and source installation. My original plan with regards to ebuild testing is to use an existing solution called ebuildtester [1], which is a program that can install a specified package in a stage3 Docker container. After doing some research on it, I found that it cannot satisfy some of my ebuild testing requirements for the Kotlin ebuilds I had created in the previous weeks at least. I would like to fully customize USE flag settings just like how I could specify USE flag settings in /etc/portage/package.use, but ebuildtester only supports a global USE flag setting and USE configuration for the package being tested itself, so the granularity is too coarse. My Kotlin ebuilds need to be installed with multiple emerge commands, which is quite uncommon for ebuilds to be honest, but with ebuildtester, it does not seem to be possible to run more than one emerge command. However, I still liked the idea of testing ebuild installation in a Docker container, so based on ebuildtester's concept, I created my own tool that tests ebuilds in a container: ebuild-commander [2]. This tool does not take an atom to be installed as input; instead, it takes a list of commands to execute in the container (hence the name "ebuild-commander"). It also supports copying a specified directory's contents to the container's /etc/portage, allowing full customization on Portage settings in ebuild tests. ebuild-commander is like a rewrite of ebuildtester strived to allow full control over the ebuild test execution with a simpler interface. Automated and periodic ebuild testing with ebuild-commander has now been set up for my fork of the Spark overlay [3]. I have only created two test cases for my Kotlin ebuilds at this point, but more tests can be added later quickly and easily. At this point, I am planning to implement a mechanism to automatically compute a minimal set of packages that need to be specified in arguments to emerge in order to install all packages in the Spark overlay at least once. Because a test case can take a very long time to run, the tests are configured to run once a day only instead of on every new commit. That is all for this week. See you in the next weekly report! Have a good week, Leo [1]: https://github.com/nicolasbock/ebuildtester [2]: https://github.com/Leo3418/ebuild-commander [3]: https://github.com/Leo3418/spark-overlay/actions/workflows/docker.yml