From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 681EF138359 for ; Tue, 3 Nov 2020 03:44:25 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 0F7F2E0A9B; Tue, 3 Nov 2020 03:44:24 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 8E062E0A9A; Tue, 3 Nov 2020 03:44:23 +0000 (UTC) Received: by mail-ed1-f43.google.com with SMTP id l24so16779343edj.8; Mon, 02 Nov 2020 19:44:22 -0800 (PST) X-Gm-Message-State: AOAM531j10kCS6HSqGQM7ZqyGl+UAcdG2dQKKUEUUPgkGEuXXI+wCN0m jwav9jlEd9zWr8W5CuHyOoM0/UorH5teKBYrO80= X-Google-Smtp-Source: ABdhPJxE3tjb4zY9UKOKtbAH87pvP6piyLijncVmRFDTphC4ViBTGBqZZeUqSZ0+VRwWdbwoIZOjte6qtlKT46ZCXXY= X-Received: by 2002:aa7:d34e:: with SMTP id m14mr8520868edr.42.1604375059083; Mon, 02 Nov 2020 19:44:19 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-catalyst@lists.gentoo.org Reply-to: gentoo-catalyst@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 From: Matt Turner Date: Mon, 2 Nov 2020 22:44:07 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: [gentoo-catalyst] catalyst changes for improving automation To: gentoo-catalyst@lists.gentoo.org, gentoo-releng@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Archives-Salt: 024abc6e-f038-4009-b5f0-e78229b13a3f X-Archives-Hash: 2017fd49eab7dc933e88a658cff23e01 The catalyst-auto automation scripts live in a repo separate from catalyst. That increases the difficulty of changing catalyst's interface, and it doesn't seem to offer any advantages otherwise. (Keeping build specs in a separate repo allows them to be updated independent of catalyst and that is valuable). Additionally, since the primary way catalyst is used is via this automation, it makes sense to support this workflow in catalyst directly. But to get there, there are some changes to catalyst that I think are improvements on their own and simplify the path to integrating automation capabilities directly into catalyst. That's what I'd like to discuss here. I'd like to: 1) Replace the custom .spec file format with TOML 2) Combine .spec file sequences (e.g., stage1 -> stage2 -> stage3 -> livecd-stage1 -> livecd-stage2) into a single file. I suggest naming this a ".build" file. This will also allow us to remove the redundant information that currently has to be specified in stage1.spec, stage2.spec, stage3.spec, like rel_type, version, profile, etc. It also means that we remove the nonsensical ability to change settings from one stage to the next that should not change (e.g., rel_type, version). 3) Add ability to denote which stage builds produce artifacts we care about (and want to save and/or upload) and which are just temporary. If they're temporary (e.g., a stage1 build) we can delete the artifact after the build sequence has no further use of it, and we can skip compressing the result, etc. To that end, I'm starting by figuring out what I would like the new spec file format to look like. Below are some open questions and then a strawman new-style spec file. =E2=80=A2 The .spec files in releng.git are really templates that are not directly usable without sed'ing @REPO_DIR@ and @TIMESTAMP@. It would be nice if they were directly usable as that would reduce confusion from users. =E2=80=A2 Can we make them directly usable? =E2=80=A2 Perhaps we can make catalyst handle the replacements directly? =E2=80=A2 Calculating @TIMESTAMP@ is trivially doable=E2=80=94we do it = today (see below) =E2=80=A2 We could configure @REPO_DIR@ in catalyst.conf and let cataly= st do the replacement, or we could just make the field relative to some path specified in catalyst.conf? =E2=80=A2 In the current automation scripts, we generate a value for @TIMESTAMP@ from the git HEAD used in creating the snapshot. =E2=80=A2 Would be nice to remove the dependence on the squashfs snapshot generation=E2=80=94not difficult to do =E2=80=A2 Can we generate and upload a .build file with replacements done t= o make stage builds more easily reproducible? Seems easy. Thoughts? Are there advantages to the current system I'm not considering? ppc.build: [build] arch =3D "ppc" subarch =3D "ppc" version =3D "@TIMESTAMP@" rel_type =3D "default" profile =3D "default/linux/powerpc/ppc32/17.0" source_subpath =3D "default/stage3-ppc-latest" portage_confdir =3D "@REPO_DIR@/releases/portage/stages" [build.snapshot] snapshot =3D "@TIMESTAMP@" [build.stage1] update_seed =3D "yes" update_seed_command =3D "--update --deep --newuse @world" [build.stage2] skip =3D true # Stage3 will be built directly from stage1 [build.stage3] compression =3D "xz" [build.iso] portage_confdir =3D "@REPO_DIR@/releases/portage/isos" [build.iso.stage1] use =3D [ "compile-locales", "fbcon", "ipv6", "livecd", "modules", "ncurses", "nls", "nptl", "pam", "readline", "socks5", "ssl", "static-libs", "unicode", "xml", ] packages =3D [ "app-admin/pwgen", "app-admin/syslog-ng", "app-arch/unzip", "app-crypt/gnupg", "app-laptop/pbbuttonsd", "app-misc/livecd-tools", "app-misc/screen", "app-portage/mirrorselect", "app-text/wgetpaste", "net-analyzer/tcptraceroute", "net-analyzer/traceroute", "net-dialup/mingetty", "net-dialup/pptpclient", "net-dialup/rp-pppoe", "net-fs/cifs-utils", "net-fs/nfs-utils", "net-irc/irssi", "net-misc/dhcpcd", "net-misc/iputils", "net-misc/ntp", "net-misc/openssh", "net-misc/rdate", "net-misc/rsync", "net-wireless/wireless-tools", "net-wireless/wpa_supplicant", "sys-apps/busybox", "sys-apps/ethtool", "sys-apps/fxload", "sys-apps/hdparm", "sys-apps/hwsetup", "sys-apps/ibm-powerpc-utils", "sys-apps/iproute2", "sys-apps/lm-sensors", "sys-apps/memtester", "sys-apps/pcmciautils", "sys-apps/powerpc-utils", "sys-apps/sdparm", "sys-block/parted", "sys-boot/grub", "sys-firmware/b43-firmware", "sys-firmware/b43legacy-firmware", "sys-fs/btrfs-progs", "sys-fs/dosfstools", "sys-fs/e2fsprogs", "sys-fs/hfsplusutils", "sys-fs/hfsutils", "sys-fs/iprutils", "sys-fs/jfsutils", "sys-fs/lvm2", "sys-fs/mac-fdisk", "sys-fs/mdadm", "sys-fs/ntfs3g", "sys-fs/reiserfsprogs", "sys-fs/xfsprogs", "sys-libs/gpm", "www-client/links", ] [build.iso.stage2] fstype =3D "squashfs" iso =3D "/var/tmp/catalyst/builds/default/install-powerpc-minimal-@TIMESTAM= P@.iso" type =3D "gentoo-release-minimal" rcadd =3D [ ["pbbuttonsd", "default"], ] unmerge =3D [ "app-admin/eselect", "app-admin/eselect-ctags", "app-admin/eselect-vi", "app-admin/perl-cleaner", "app-admin/python-updater", "app-arch/cpio", "dev-libs/gmp", "dev-libs/libxml2", "dev-libs/mpfr", "dev-libs/popt", "dev-python/pycrypto", "dev-util/pkgconfig", "perl-core/PodParser", "perl-core/Test-Harness", "sys-apps/debianutils", "sys-apps/diffutils", "sys-apps/groff", "sys-apps/man-db", "sys-apps/man-pages", "sys-apps/miscfiles", "sys-apps/sandbox", "sys-apps/texinfo", "sys-devel/autoconf", "sys-devel/autoconf-wrapper", "sys-devel/automake", "sys-devel/automake-wrapper", "sys-devel/binutils", "sys-devel/binutils-config", "sys-devel/bison", "sys-devel/flex", "sys-devel/gcc", "sys-devel/gcc-config", "sys-devel/gettext", "sys-devel/gnuconfig", "sys-devel/libtool", "sys-devel/m4", "sys-devel/make", "sys-devel/patch", "sys-libs/db", "sys-libs/gdbm", "sys-libs/libkudzu", "sys-kernel/genkernel", "sys-kernel/linux-headers", ] empty =3D [ "/boot", "/boot/initr*", "/boot/kernel*", "/boot/System*", "/etc/*-", "/etc/cron.daily", "/etc/cron.hourly", "/etc/cron.monthly", "/etc/cron.weekly", "/etc/default/audioctl", "/etc/dispatch-conf.conf", "/etc/env.d/05binutils", "/etc/env.d/05gcc", "/etc/etc-update.conf", "/etc/genkernel.conf", "/etc/hosts.bck", "/etc/issue*", "/etc/logrotate.d", "/etc/make.conf*", "/etc/make.globals", "/etc/make.profile", "/etc/man.conf", "/etc/modules.autoload.d", "/etc/*.old", "/etc/resolv.conf", "/etc/runlevels/single", "/etc/skel", "/lib64/dev-state", "/lib64/udev-state", "/lib*/*.a", "/lib*/cpp", "/lib/dev-state", "/lib*/*.la", "/lib/udev-state", "/root/.bash_history", "/root/.ccache", "/root/.viminfo", "/sbin/fsck.cramfs", "/sbin/fsck.minix", "/sbin/mkfs.bfs", "/sbin/mkfs.cramfs", "/sbin/mkfs.minix", "/sbin/*.static", "/tmp", "/usr/bin/addr2line", "/usr/bin/ar", "/usr/bin/as", "/usr/bin/audioctl", "/usr/bin/c++*", "/usr/bin/cc", "/usr/bin/cjpeg", "/usr/bin/cpp", "/usr/bin/djpeg", "/usr/bin/ebuild", "/usr/bin/egencache", "/usr/bin/elftoaout", "/usr/bin/emerge", "/usr/bin/emerge-webrsync", "/usr/bin/emirrordist", "/usr/bin/f77", "/usr/bin/g++*", "/usr/bin/g77", "/usr/bin/gcc*", "/usr/bin/genkernel", "/usr/bin/gprof", "/usr/bin/jpegtran", "/usr/bin/ld", "/usr/bin/libpng*", "/usr/bin/nm", "/usr/bin/objcopy", "/usr/bin/objdump", "/usr/bin/piggyback*", "/usr/bin/portageq", "/usr/bin/powerpc64-unknown-linux-gnu-*", "/usr/bin/powerpc-unknown-linux-gnu-*", "/usr/bin/ranlib", "/usr/bin/readelf", "/usr/bin/repoman", "/usr/bin/size", "/usr/bin/strings", "/usr/bin/strip", "/usr/bin/tbz2tool", "/usr/bin/xpak", "/usr/bin/yacc", "/usr/diet/include", "/usr/diet/man", "/usr/include", "/usr/lib64/awk", "/usr/lib64/ccache", "/usr/lib64/gcc-config", "/usr/lib64/gconv", "/usr/lib64/nfs", "/usr/lib64/perl5/site_perl", "/usr/lib64/portage", "/usr/lib64/python*/test", "/usr/lib64/X11/config", "/usr/lib64/X11/doc", "/usr/lib64/X11/etc", "/usr/lib*/*.a", "/usr/lib*/gcc-lib/*/*/libgcj*", "/usr/lib*/*.la", "/usr/lib*/perl5/site_perl", "/usr/local", "/usr/portage", "/usr/powerpc64-unknown-linux-gnu", "/usr/powerpc-unknown-linux-gnu", "/usr/sbin/archive-conf", "/usr/sbin/dispatch-conf", "/usr/sbin/emaint", "/usr/sbin/env-update", "/usr/sbin/etc-update", "/usr/sbin/fb*", "/usr/sbin/fixpackages", "/usr/sbin/quickpkg", "/usr/sbin/regenworld", "/usr/share/aclocal", "/usr/share/baselayout", "/usr/share/binutils-data", "/usr/share/consolefonts/1*", "/usr/share/consolefonts/7*", "/usr/share/consolefonts/8*", "/usr/share/consolefonts/9*", "/usr/share/consolefonts/a*", "/usr/share/consolefonts/A*", "/usr/share/consolefonts/c*", "/usr/share/consolefonts/C*", "/usr/share/consolefonts/dr*", "/usr/share/consolefonts/E*", "/usr/share/consolefonts/g*", "/usr/share/consolefonts/G*", "/usr/share/consolefonts/i*", "/usr/share/consolefonts/k*", "/usr/share/consolefonts/l*", "/usr/share/consolefonts/L*", "/usr/share/consolefonts/M*", "/usr/share/consolefonts/partialfonts", "/usr/share/consolefonts/r*", "/usr/share/consolefonts/R*", "/usr/share/consolefonts/s*", "/usr/share/consolefonts/t*", "/usr/share/consolefonts/v*", "/usr/share/consoletrans", "/usr/share/dict", "/usr/share/doc", "/usr/share/emacs", "/usr/share/et", "/usr/share/gcc-data", "/usr/share/genkernel", "/usr/share/gettext", "/usr/share/glib-2.0", "/usr/share/gnuconfig", "/usr/share/gtk-doc", "/usr/share/i18n", "/usr/share/info", "/usr/share/lcms", "/usr/share/libtool", "/usr/share/locale", "/usr/share/man", "/usr/share/misc/*.old", "/usr/share/rfc", "/usr/share/ss", "/usr/share/state", "/usr/share/texinfo", "/usr/share/unimaps", "/usr/share/zoneinfo", "/usr/src", "/var/cache", "/var/empty", "/var/lib/portage", "/var/log", "/var/spool", "/var/state", "/var/tmp", ] # Just for bootloader ordering purposes kernel =3D ["ppc64", "ppc32", "ibmpower",] [build.iso.stage2.kernel.ibmpower] sources =3D "sys-kernel/gentoo-sources" config =3D "../../../kconfig/powerpc/installcd-ibm-4.19.config" console =3D ["ttyS0,9600", "hvc0", "hvsi0",] gk_kernargs =3D [ "--kernel-cc=3D'gcc -m64'", "--kernel-ld=3D'ld -m elf64ppc'", "--kernel-as=3D'as -a64'", ] [build.iso.stage2.kernel.ppc64] sources =3D "sys-kernel/gentoo-sources" config =3D "../../../kconfig/powerpc/ppc64.config" console =3D ["ttyS0,57600",] gk_kernargs =3D [ "--kernel-cc=3D'gcc -m64'", "--kernel-ld=3D'ld -m elf64ppc'", "--kernel-as=3D'as -a64'", ] [build.iso.stage2.kernel.ppc32] sources =3D "sys-kernel/gentoo-sources" config =3D "../../../kconfig/powerpc/ppc32.config"