Re: [gentoo-catalyst] catalyst changes for improving automation

[Matt Turner]

On Tue, Nov 3, 2020 at 5:56 AM Daniel Cordero <gentoo.catalyst@××××.ws> wrote:
>
> On Mon, Nov 02, 2020 at 10:44:07PM -0500, Matt Turner wrote:
> > The catalyst-auto automation scripts live in a repo separate from
> > catalyst. That increases the difficulty of changing catalyst's
> > interface, and it doesn't seem to offer any advantages otherwise.
> > (Keeping build specs in a separate repo allows them to be updated
> > independent of catalyst and that is valuable). Additionally, since the
> > primary way catalyst is used is via this automation, it makes sense to
> > support this workflow in catalyst directly.
> >
>
> What would be more heavily impacted are those users who may not already
> have infra set up to do builds or just starting out using catalyst for
> the first time and haven't written their own automation.
>
> I suggest prioritising the collection of up-to-date documentation,
> especially regarding running catalyst manually, since it'll be
> completely different to the literature that's currently out there.

I'm a bit unsure what you mean. Do you suggest prioritizing
documenting the current method of running catalyst before changing it?

> > But to get there, there are some changes to catalyst that I think are
> > improvements on their own and simplify the path to integrating
> > automation capabilities directly into catalyst. That's what I'd like
> > to discuss here.
> >
> > I'd like to:
> >
> >  1) Replace the custom .spec file format with TOML
> >
>
> Fine. Aside from the extra quotes and commas, I'd be happy with any well
> defined format that can handle strings and lists properly.
>
> >  2) Combine .spec file sequences (e.g., stage1 -> stage2 -> stage3 ->
> > livecd-stage1 -> livecd-stage2) into a single file. I suggest naming
> > this a ".build" file. This will also allow us to remove the redundant
> > information that currently has to be specified in stage1.spec,
> > stage2.spec, stage3.spec, like rel_type, version, profile, etc. It
> > also means that we remove the nonsensical ability to change settings
> > from one stage to the next that should not change (e.g., rel_type,
> > version).
> >
>
> How would a target that depends on a different rel_type work? Forks in
> the dependency tree.
>
> >  3) Add ability to denote which stage builds produce artifacts we care
> > about (and want to save and/or upload) and which are just temporary.
> > If they're temporary (e.g., a stage1 build) we can delete the artifact
> > after the build sequence has no further use of it, and we can skip
> > compressing the result, etc.
> >
>
> This feature should (haven't tested) already exist - it's just not
> documented.
>
> compression_mode: rsync
> options=['seedcache']

Hah! I was completely unaware of this. Thanks.

> or don't call 'capture' and/or 'remove_chroot' in action_/finish_sequence.
>
> >
> > To that end, I'm starting by figuring out what I would like the new
> > spec file format to look like. Below are some open questions and then
> > a strawman new-style spec file.
> >
> > • The .spec files in releng.git are really templates that are not
> > directly usable without sed'ing @REPO_DIR@ and @TIMESTAMP@. It would
> > be nice if they were directly usable as that would reduce confusion
> > from users.
> >   • Can we make them directly usable?
> >   • Perhaps we can make catalyst handle the replacements directly?
> >     • Calculating @TIMESTAMP@ is trivially doable—we do it today (see below)
>
> Maybe a strftime() template, or even fstring-like tokens?
> (e.g. "{year}-{month}-{day}")

One goal I have is to make it more transparent what is actually in a
particular stage tarball or ISO and along with that to make it easier
to reproduce the result.

Obviously we'll want to keep the ability to specify a particular
version, as you describe, but I think for Gentoo releases we will want
to continue using a timestamp that's unambiguously tied to the git
SHA1 of gentoo.git as is possible.

> >     • We could configure @REPO_DIR@ in catalyst.conf and let catalyst
> > do the replacement, or we could just make the field relative to some
> > path specified in catalyst.conf?
> >
>
> While nice to have, I don't agree with locking users into a particular
> repository layout.

Can you explain what you mean? I don't know how what I said would
require a particular repository layout.

Perhaps you're confused by the @REPO_DIR@ name? It is the path to the
releng.git repository (containing the .specs and the /etc/portage/
files) on the build machine and is not in any way connected with the
ebuild repositories.

The name predates my involvement, so don't blame me :)

> > • In the current automation scripts, we generate a value for
> > @TIMESTAMP@ from the git HEAD used in creating the snapshot.
> >   • Would be nice to remove the dependence on the squashfs snapshot
> > generation—not difficult to do
> >
>
> I have no comment on this.
>
> > • Can we generate and upload a .build file with replacements done to
> > make stage builds more easily reproducible? Seems easy.
> >
>
> These can just be artifacts from the build.

Yes, that's what I'm thinking too.