From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id DD52A138359 for ; Fri, 30 Oct 2020 16:14:45 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 1939AE0837; Fri, 30 Oct 2020 16:14:45 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id DF0A1E0837 for ; Fri, 30 Oct 2020 16:14:44 +0000 (UTC) Received: by mail-ed1-f47.google.com with SMTP id a71so1865066edf.9 for ; Fri, 30 Oct 2020 09:14:13 -0700 (PDT) X-Gm-Message-State: AOAM530xFtVdvXuaZAyx3pyTkAT1ohTLSZE9J1CIyCPp84HepKsvoWXv EvtNjYUeeFG3dUHwjP9+aQCwQMqAxftI9mvkS6w= X-Google-Smtp-Source: ABdhPJwTVtVMqHmpm8sKazH2sRm8ERli389XJsjUkjP1ZZIgefRAVPf3rgohtoRqVnjmC+HwWs9beb5tk/Pik3HHl8I= X-Received: by 2002:aa7:c792:: with SMTP id n18mr3198625eds.209.1604074450966; Fri, 30 Oct 2020 09:14:10 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-catalyst@lists.gentoo.org Reply-to: gentoo-catalyst@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <914cce4739709511771f27bed773c7e4e01c3c9a.camel@rohde-schwarz.com> In-Reply-To: From: Matt Turner Date: Fri, 30 Oct 2020 12:13:58 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [Newsletter] Re: [gentoo-catalyst] [PATCH 2/2] Move from PORTDIR_OVERLAY to repos.conf To: gentoo-catalyst@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" X-Archives-Salt: 40a544c0-59d0-456c-974a-a3348960a708 X-Archives-Hash: f54a5f96e1e9a0bd145ef981188d3b9e On Sun, Oct 18, 2020 at 9:58 AM Felix Bier wrote: > > Am Samstag, den 17.10.2020, 13:11 -0700 schrieb Matt Turner: > > On Sat, Oct 17, 2020 at 12:00 PM Felix Bier > > wrote: > > > > > > This commit fixes the following issues: > > > > > > * The PORTDIR_OVERLAY variable has been deprecated by Gentoo. > > > > > > With this commit, the variable is no longer written to the > > > generated make.conf. Instead, a config file > > > /etc/portage/repos.conf/.conf > > > is generated for each overlay. The repo name is read from the > > > overlay using the portage API. Internally, portage parses > > > metadata/layout.conf and profiles/repo_name to obtain the name. > > > > > > References: > > > https://wiki.gentoo.org/wiki//etc/portage/make.conf > > > https://wiki.gentoo.org/wiki//etc/portage/repos.conf > > > > > > * All overlays were copied into the same target directory. If the > > > same file name occurred in multiple overlays, the last overlay > > > would overwrite all previous files with this name. In > > > particular, > > > only the metadata/layout.conf of the last overlay was retained, > > > so it was not possible to reference the other overlays e.g. via > > > the masters entry in the layout.conf or the portage-2 syntax > > > for specifying a parent profile from another overlay. Also, > > > this created problems when the overlays contained ebuilds > > > for the same package, but with differing versions, because > > > after copying, the target directory contained both versions of > > > the > > > ebuild but only the manifest file of the last overlay. > > > > > > With this commit, each overlay is copied into a separate > > > sub-directory, e.g. /var/gentoo/repos/local//. > > > This directory is referenced via the location entry in the > > > generated /etc/portage/repos.conf/.conf. > > > --- > > > > Hello, > > > > Thank you for the patches. I'm happy to see them. > > > > I cannot apply the patches however. This one in particular is badly > > line wrapped by your mail client. I tried fixing it up, but either > > failed or don't know what this is supposed to apply to. > > > > It looks like the intention is for this to apply to the > > catalyst-3.0-stable branch since it discusses copying overlays into a > > subdirectory, rather than any mention of squashfs snapshots. > > > > I really don't want new feature work on the catalyst-3.0-stable > > branch, for example. If that is the target, then please consider > > rebasing the work onto the master branch. > > > > Matt > > Hello Matt, > thank you for the initial review. > > I'm sorry for the formatting issues, I will check my mail client's > format settings and resend the patches. > > The patches are intendend for the master branch. I agree that this > feature should not be in the stable branch. > > My understanding is that in the master branch, the squashfs is only > used for the main repository, whereas the overlays are still copied as > directories (in the StageBase.portage_overlay() function). > > This patch only removes PORTDIR_OVERLAY, not PORTDIR, so it should not > be a problem that the main repository is squashed. I kept the helper > functions that this patch adds general, so that they can be reused for > a later PORTDIR removal, but I would prefer to do that in a separate > step. Right. I guess I was expecting a transition from PORTDIR + PORTDIR_OVERLAY to the modern just-a-bunch-of-repos repos.conf system. That's totally fine to not make the leap all at once. There's actually a patch in https://bugs.gentoo.org/560518 that creates repos.conf, if you're interested in any follow on work :) > I don't know if there are plans to also switch the overlays to > squashfs. My intuition would be that the overlays would usually be much > smaller than the main repository, so squashing them would have less > benefit. Even if there is a plan to switch the overlays to squashfs (or > bind-mounting, which I would prefer), I think my patch would still be > helpful for that, since it would not be possible to create multiple > mounts with the same target directory (such as /var/db/repo/local). So > creating subdirectories (/var/db/repo/local/overlay1, > /var/db/repo/local/overlay2 etc.) would still be needed. An additional advantage of squashfs snapshots is that they're snapshots. In that way, we should be able to reproduce a build from exactly the same inputs. You're right that the other advantages of squashfs snapshots (smaller size, not copying 1GB of small files, quick cleanup, etc) don't really apply to the same degree to small overlays.