From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id D96FC138359 for ; Tue, 3 Nov 2020 10:56:20 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 163C7E0A93; Tue, 3 Nov 2020 10:56:20 +0000 (UTC) Received: from rs234.mailgun.us (rs234.mailgun.us [209.61.151.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id A338AE0A95 for ; Tue, 3 Nov 2020 10:56:14 +0000 (UTC) DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=xxoo.ws; q=dns/txt; s=mx; t=1604400979; h=In-Reply-To: Content-Transfer-Encoding: Content-Type: MIME-Version: References: Message-ID: Subject: Cc: To: From: Date: Sender; bh=3UZcckU+l3WQY0O7bgMfGt9OwMzGewEktQm+QBtcITU=; b=H1fZylcgDD5ZkMw73JcwvJ2OOExo3xvzPKGsQ+slAHoF6y1BuX8gnewc2+5qyqvAW4yGdj1o kr01g8Pl4FWW3Hpw8Lw9MiGLUl8YFSvKTROKoC8zUDXfIddpU/cNv8TNI+Ykx0oHdyaUXIWj mrKOWeVfdDCZDvzVzBI+33CvFwg= X-Mailgun-Sending-Ip: 209.61.151.234 X-Mailgun-Sid: WyIxYmU3MyIsICJnZW50b28tY2F0YWx5c3RAbGlzdHMuZ2VudG9vLm9yZyIsICJlYTQwNTYiXQ== Received: from mail.0xdc.io (ip-54-37-0.eu [54.37.0.172]) by smtp-out-n01.prod.us-west-2.postgun.com with SMTP id 5fa137060ce128468b83f327 (version=TLS1.3, cipher=TLS_AES_128_GCM_SHA256); Tue, 03 Nov 2020 10:55:01 GMT Sender: gentoo.catalyst@xxoo.ws Received: from dysnomia (2.b.3.7.5.1.f.d.2.3.c.4.7.7.0.a.0.a.4.7.9.0.c.0.0.b.8.0.1.0.0.2.ip6.arpa [IPv6:2001:8b0:c09:74a0:a077:4c32:df15:73b2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.0xdc.io (Postfix) with ESMTPSA id 96765102D60; Tue, 3 Nov 2020 11:01:25 +0000 (UTC) Date: Tue, 3 Nov 2020 10:54:56 +0000 From: Daniel Cordero To: gentoo-catalyst@lists.gentoo.org Cc: gentoo-releng@lists.gentoo.org Subject: Re: [gentoo-catalyst] catalyst changes for improving automation Message-ID: <20201103105456.GA3468@dysnomia.localdomain> References: Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-catalyst@lists.gentoo.org Reply-to: gentoo-catalyst@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Archives-Salt: 07ed9da3-d639-4d4b-95fe-0bb3c7d220dd X-Archives-Hash: ddab08d41dec815832a2dbb147ecb743 On Mon, Nov 02, 2020 at 10:44:07PM -0500, Matt Turner wrote: > The catalyst-auto automation scripts live in a repo separate from > catalyst. That increases the difficulty of changing catalyst's > interface, and it doesn't seem to offer any advantages otherwise. > (Keeping build specs in a separate repo allows them to be updated > independent of catalyst and that is valuable). Additionally, since the > primary way catalyst is used is via this automation, it makes sense to > support this workflow in catalyst directly. > What would be more heavily impacted are those users who may not already have infra set up to do builds or just starting out using catalyst for the first time and haven't written their own automation. I suggest prioritising the collection of up-to-date documentation, especially regarding running catalyst manually, since it'll be completely different to the literature that's currently out there. > But to get there, there are some changes to catalyst that I think are > improvements on their own and simplify the path to integrating > automation capabilities directly into catalyst. That's what I'd like > to discuss here. > > I'd like to: > > 1) Replace the custom .spec file format with TOML > Fine. Aside from the extra quotes and commas, I'd be happy with any well defined format that can handle strings and lists properly. > 2) Combine .spec file sequences (e.g., stage1 -> stage2 -> stage3 -> > livecd-stage1 -> livecd-stage2) into a single file. I suggest naming > this a ".build" file. This will also allow us to remove the redundant > information that currently has to be specified in stage1.spec, > stage2.spec, stage3.spec, like rel_type, version, profile, etc. It > also means that we remove the nonsensical ability to change settings > from one stage to the next that should not change (e.g., rel_type, > version). > How would a target that depends on a different rel_type work? Forks in the dependency tree. > 3) Add ability to denote which stage builds produce artifacts we care > about (and want to save and/or upload) and which are just temporary. > If they're temporary (e.g., a stage1 build) we can delete the artifact > after the build sequence has no further use of it, and we can skip > compressing the result, etc. > This feature should (haven't tested) already exist - it's just not documented. compression_mode: rsync options=['seedcache'] or don't call 'capture' and/or 'remove_chroot' in action_/finish_sequence. > > To that end, I'm starting by figuring out what I would like the new > spec file format to look like. Below are some open questions and then > a strawman new-style spec file. > > • The .spec files in releng.git are really templates that are not > directly usable without sed'ing @REPO_DIR@ and @TIMESTAMP@. It would > be nice if they were directly usable as that would reduce confusion > from users. > • Can we make them directly usable? > • Perhaps we can make catalyst handle the replacements directly? > • Calculating @TIMESTAMP@ is trivially doable—we do it today (see below) Maybe a strftime() template, or even fstring-like tokens? (e.g. "{year}-{month}-{day}") > • We could configure @REPO_DIR@ in catalyst.conf and let catalyst > do the replacement, or we could just make the field relative to some > path specified in catalyst.conf? > While nice to have, I don't agree with locking users into a particular repository layout. > • In the current automation scripts, we generate a value for > @TIMESTAMP@ from the git HEAD used in creating the snapshot. > • Would be nice to remove the dependence on the squashfs snapshot > generation—not difficult to do > I have no comment on this. > • Can we generate and upload a .build file with replacements done to > make stage builds more easily reproducible? Seems easy. > These can just be artifacts from the build.