Re: [gentoo-catalyst] catalyst changes for improving automation

[Daniel Cordero]

On Mon, Nov 02, 2020 at 10:44:07PM -0500, Matt Turner wrote:
> The catalyst-auto automation scripts live in a repo separate from
> catalyst. That increases the difficulty of changing catalyst's
> interface, and it doesn't seem to offer any advantages otherwise.
> (Keeping build specs in a separate repo allows them to be updated
> independent of catalyst and that is valuable). Additionally, since the
> primary way catalyst is used is via this automation, it makes sense to
> support this workflow in catalyst directly.
>

What would be more heavily impacted are those users who may not already
have infra set up to do builds or just starting out using catalyst for
the first time and haven't written their own automation.

I suggest prioritising the collection of up-to-date documentation,
especially regarding running catalyst manually, since it'll be
completely different to the literature that's currently out there.


> But to get there, there are some changes to catalyst that I think are
> improvements on their own and simplify the path to integrating
> automation capabilities directly into catalyst. That's what I'd like
> to discuss here.
>
> I'd like to:
>
>  1) Replace the custom .spec file format with TOML
>

Fine. Aside from the extra quotes and commas, I'd be happy with any well
defined format that can handle strings and lists properly.

>  2) Combine .spec file sequences (e.g., stage1 -> stage2 -> stage3 ->
> livecd-stage1 -> livecd-stage2) into a single file. I suggest naming
> this a ".build" file. This will also allow us to remove the redundant
> information that currently has to be specified in stage1.spec,
> stage2.spec, stage3.spec, like rel_type, version, profile, etc. It
> also means that we remove the nonsensical ability to change settings
> from one stage to the next that should not change (e.g., rel_type,
> version).
>

How would a target that depends on a different rel_type work? Forks in
the dependency tree.

>  3) Add ability to denote which stage builds produce artifacts we care
> about (and want to save and/or upload) and which are just temporary.
> If they're temporary (e.g., a stage1 build) we can delete the artifact
> after the build sequence has no further use of it, and we can skip
> compressing the result, etc.
>

This feature should (haven't tested) already exist - it's just not
documented.

compression_mode: rsync
options=['seedcache']

or don't call 'capture' and/or 'remove_chroot' in action_/finish_sequence.

>
> To that end, I'm starting by figuring out what I would like the new
> spec file format to look like. Below are some open questions and then
> a strawman new-style spec file.
>
> • The .spec files in releng.git are really templates that are not
> directly usable without sed'ing @REPO_DIR@ and @TIMESTAMP@. It would
> be nice if they were directly usable as that would reduce confusion
> from users.
>   • Can we make them directly usable?
>   • Perhaps we can make catalyst handle the replacements directly?
>     • Calculating @TIMESTAMP@ is trivially doable—we do it today (see below)

Maybe a strftime() template, or even fstring-like tokens?
(e.g. "{year}-{month}-{day}")

>     • We could configure @REPO_DIR@ in catalyst.conf and let catalyst
> do the replacement, or we could just make the field relative to some
> path specified in catalyst.conf?
>

While nice to have, I don't agree with locking users into a particular
repository layout.

> • In the current automation scripts, we generate a value for
> @TIMESTAMP@ from the git HEAD used in creating the snapshot.
>   • Would be nice to remove the dependence on the squashfs snapshot
> generation—not difficult to do
>

I have no comment on this.

> • Can we generate and upload a .build file with replacements done to
> make stage builds more easily reproducible? Seems easy.
>

These can just be artifacts from the build.