public inbox for gentoo-catalyst@lists.gentoo.org
 help / color / mirror / Atom feed
From: Matt Turner <mattst88@gentoo.org>
To: gentoo-catalyst@lists.gentoo.org
Cc: gentoo-releng@lists.gentoo.org
Subject: Re: [gentoo-catalyst] catalyst changes for improving automation
Date: Tue, 3 Nov 2020 13:19:51 -0500	[thread overview]
Message-ID: <CAEdQ38GLe0o9bnRVauuUkAjkCtmqb8MveZ4HFpmFvg3EgORWZA@mail.gmail.com> (raw)
In-Reply-To: <20201103105456.GA3468@dysnomia.localdomain>

On Tue, Nov 3, 2020 at 5:56 AM Daniel Cordero <gentoo.catalyst@xxoo.ws> wrote:
>
> On Mon, Nov 02, 2020 at 10:44:07PM -0500, Matt Turner wrote:
> > The catalyst-auto automation scripts live in a repo separate from
> > catalyst. That increases the difficulty of changing catalyst's
> > interface, and it doesn't seem to offer any advantages otherwise.
> > (Keeping build specs in a separate repo allows them to be updated
> > independent of catalyst and that is valuable). Additionally, since the
> > primary way catalyst is used is via this automation, it makes sense to
> > support this workflow in catalyst directly.
> >
>
> What would be more heavily impacted are those users who may not already
> have infra set up to do builds or just starting out using catalyst for
> the first time and haven't written their own automation.
>
> I suggest prioritising the collection of up-to-date documentation,
> especially regarding running catalyst manually, since it'll be
> completely different to the literature that's currently out there.

I'm a bit unsure what you mean. Do you suggest prioritizing
documenting the current method of running catalyst before changing it?

> > But to get there, there are some changes to catalyst that I think are
> > improvements on their own and simplify the path to integrating
> > automation capabilities directly into catalyst. That's what I'd like
> > to discuss here.
> >
> > I'd like to:
> >
> >  1) Replace the custom .spec file format with TOML
> >
>
> Fine. Aside from the extra quotes and commas, I'd be happy with any well
> defined format that can handle strings and lists properly.
>
> >  2) Combine .spec file sequences (e.g., stage1 -> stage2 -> stage3 ->
> > livecd-stage1 -> livecd-stage2) into a single file. I suggest naming
> > this a ".build" file. This will also allow us to remove the redundant
> > information that currently has to be specified in stage1.spec,
> > stage2.spec, stage3.spec, like rel_type, version, profile, etc. It
> > also means that we remove the nonsensical ability to change settings
> > from one stage to the next that should not change (e.g., rel_type,
> > version).
> >
>
> How would a target that depends on a different rel_type work? Forks in
> the dependency tree.
>
> >  3) Add ability to denote which stage builds produce artifacts we care
> > about (and want to save and/or upload) and which are just temporary.
> > If they're temporary (e.g., a stage1 build) we can delete the artifact
> > after the build sequence has no further use of it, and we can skip
> > compressing the result, etc.
> >
>
> This feature should (haven't tested) already exist - it's just not
> documented.
>
> compression_mode: rsync
> options=['seedcache']

Hah! I was completely unaware of this. Thanks.

> or don't call 'capture' and/or 'remove_chroot' in action_/finish_sequence.
>
> >
> > To that end, I'm starting by figuring out what I would like the new
> > spec file format to look like. Below are some open questions and then
> > a strawman new-style spec file.
> >
> > • The .spec files in releng.git are really templates that are not
> > directly usable without sed'ing @REPO_DIR@ and @TIMESTAMP@. It would
> > be nice if they were directly usable as that would reduce confusion
> > from users.
> >   • Can we make them directly usable?
> >   • Perhaps we can make catalyst handle the replacements directly?
> >     • Calculating @TIMESTAMP@ is trivially doable—we do it today (see below)
>
> Maybe a strftime() template, or even fstring-like tokens?
> (e.g. "{year}-{month}-{day}")

One goal I have is to make it more transparent what is actually in a
particular stage tarball or ISO and along with that to make it easier
to reproduce the result.

Obviously we'll want to keep the ability to specify a particular
version, as you describe, but I think for Gentoo releases we will want
to continue using a timestamp that's unambiguously tied to the git
SHA1 of gentoo.git as is possible.

> >     • We could configure @REPO_DIR@ in catalyst.conf and let catalyst
> > do the replacement, or we could just make the field relative to some
> > path specified in catalyst.conf?
> >
>
> While nice to have, I don't agree with locking users into a particular
> repository layout.

Can you explain what you mean? I don't know how what I said would
require a particular repository layout.

Perhaps you're confused by the @REPO_DIR@ name? It is the path to the
releng.git repository (containing the .specs and the /etc/portage/
files) on the build machine and is not in any way connected with the
ebuild repositories.

The name predates my involvement, so don't blame me :)

> > • In the current automation scripts, we generate a value for
> > @TIMESTAMP@ from the git HEAD used in creating the snapshot.
> >   • Would be nice to remove the dependence on the squashfs snapshot
> > generation—not difficult to do
> >
>
> I have no comment on this.
>
> > • Can we generate and upload a .build file with replacements done to
> > make stage builds more easily reproducible? Seems easy.
> >
>
> These can just be artifacts from the build.

Yes, that's what I'm thinking too.


  reply	other threads:[~2020-11-03 18:20 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-03  3:44 [gentoo-catalyst] catalyst changes for improving automation Matt Turner
2020-11-03 10:54 ` Daniel Cordero
2020-11-03 18:19   ` Matt Turner [this message]
2020-11-04 10:46     ` Daniel Cordero
2020-11-03 18:36   ` Matt Turner
2020-11-03 13:04 ` Brian Dolbec

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAEdQ38GLe0o9bnRVauuUkAjkCtmqb8MveZ4HFpmFvg3EgORWZA@mail.gmail.com \
    --to=mattst88@gentoo.org \
    --cc=gentoo-catalyst@lists.gentoo.org \
    --cc=gentoo-releng@lists.gentoo.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox