From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 3BAA4138359 for ; Thu, 29 Oct 2020 15:58:40 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 3888DE0835; Thu, 29 Oct 2020 15:58:39 +0000 (UTC) Received: from rs234.mailgun.us (rs234.mailgun.us [209.61.151.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id DCE27E0835 for ; Thu, 29 Oct 2020 15:58:38 +0000 (UTC) DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=xxoo.ws; q=dns/txt; s=mx; t=1603987118; h=Content-Type: MIME-Version: Message-ID: Subject: To: From: Date: Sender; bh=l+Zgs28t3QaY7+0+CKlPIv4cKehTJ2VjHINUiozize4=; b=KHKfQR/P5n+O8bNuO7Vv21a+uzr/xneRC5puccYeKHthy+Bt2mnnTuk3GEpdD0T3Bj14pEkZ mt1M0Mu3/03DsmUQZmdWufvJBFeyvWAslXONPZExeKPVHF35Y7Ff4zeXXmfGK7fFFHBpg9UH OSevzunq42F8SgbOW4rei20yF10= X-Mailgun-Sending-Ip: 209.61.151.234 X-Mailgun-Sid: WyIxYmU3MyIsICJnZW50b28tY2F0YWx5c3RAbGlzdHMuZ2VudG9vLm9yZyIsICJlYTQwNTYiXQ== Received: from mail.0xdc.io (ip-54-37-0.eu [54.37.0.172]) by smtp-out-n06.prod.us-west-2.postgun.com with SMTP id 5f9ae6ad95539f1aecf277c6 (version=TLS1.3, cipher=TLS_AES_128_GCM_SHA256); Thu, 29 Oct 2020 15:58:37 GMT Sender: gentoo.catalyst@xxoo.ws Received: from dysnomia (c.7.d.c.6.b.9.e.e.a.6.1.b.1.9.8.0.a.4.7.9.0.c.0.0.b.8.0.1.0.0.2.ip6.arpa [IPv6:2001:8b0:c09:74a0:891b:16ae:e9b6:cd7c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.0xdc.io (Postfix) with ESMTPSA id 35607102CE7 for ; Thu, 29 Oct 2020 16:05:00 +0000 (UTC) Date: Thu, 29 Oct 2020 15:58:32 +0000 From: Daniel Cordero To: gentoo-catalyst@lists.gentoo.org Subject: [gentoo-catalyst] [RFC PATCH] Use a 3 stage livecd build Message-ID: <20201029155832.GA875634@dysnomia.localdomain> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-catalyst@lists.gentoo.org Reply-to: gentoo-catalyst@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Archives-Salt: 9d38e2a9-41bc-46f4-9403-ee920710b6bd X-Archives-Hash: 5c490c35cb1bf78e62af3adcf63456bd From: Daniel Cordero The current (2 stage) livecd process requires a full stage3/4 tarball that can install genkernel and compile a kernel for the livecd. Once the kernel is built, packages are then unmerged to reduce the size of the CD image. This means explicity removing every unnecessary package that may be installed from the system set, or additional stage4 world sets. Instead, an embedded target could be used, or a stage1 target be abused, to create minimal images for the livecd environment. This patch moves some actions from livecd-stage1 and livecd-stage2 into a new target: livecd-stage3. livecd-stage1 is now very similar to a regular stage4, where additional packages can be installed through the package manager. This step would be optional for embedded stages without a package manager. livecd-stage2 now uses a stage3 to build a kernel and reusable "cdtar". The seed stage is not retained. livecd-stage3 takes the built kernel and modules, and a livecd-stage1/embedded/stage3/stage4 tarball. Without package manager interaction, it combines these into a livecd. Modules for the built kernels are extracted into the looped image. --- The code basically copies the existing livecd-stage1/livecd-stage2 code in the livecd-stage3 target and support scripts. I'm sure this could be cleaned up to remove the references in livecd-stage1.py/livecd-stage2.py, although I have not done that here. I did also remove the module blacklisting, X11 handling and gentoo-release-livecd/dvd file copies that was in the livecd-stage2. They'd need to be re-added. The livecd-stage3 specfile requires the 'profile:' keyword, even though it is not required. This is because it is defined as a requirement by StageBase.__init__() catalyst/targets/livecd_stage1.py | 14 +++++- catalyst/targets/livecd_stage2.py | 69 ++--------------------------- catalyst/targets/livecd_stage3.py | 61 +++++++++++++++++++++++++ targets/livecd-stage1/controller.sh | 5 +++ targets/livecd-stage3/controller.sh | 51 +++++++++++++++++++++ 5 files changed, 133 insertions(+), 67 deletions(-) create mode 100644 catalyst/targets/livecd_stage3.py create mode 100755 targets/livecd-stage3/controller.sh diff --git a/catalyst/targets/livecd_stage1.py b/catalyst/targets/livecd_stage1.py index f0b6be8b..0236eb2f 100644 --- a/catalyst/targets/livecd_stage1.py +++ b/catalyst/targets/livecd_stage1.py @@ -4,7 +4,6 @@ LiveCD stage1 target # NOTE: That^^ docstring has influence catalyst-spec(5) man page generation. from catalyst.support import normpath - from catalyst.base.stagebase import StageBase @@ -13,9 +12,14 @@ class livecd_stage1(StageBase): Builder class for LiveCD stage1. """ required_values = frozenset([ - "livecd/packages", ]) valid_values = required_values | frozenset([ + "livecd/empty", + "livecd/fsscript", + "livecd/packages", + "livecd/rm", + "livecd/root_overlay", + "livecd/unmerge", "livecd/use", ]) @@ -32,7 +36,13 @@ class livecd_stage1(StageBase): "chroot_setup", "setup_environment", "build_packages", + "root_overlay", + "fsscript", + "unmerge", + "bootloader", "unbind", + "remove", + "empty", "clean", ] self.set_completion_action_sequences() diff --git a/catalyst/targets/livecd_stage2.py b/catalyst/targets/livecd_stage2.py index 88c0d95c..b459288b 100644 --- a/catalyst/targets/livecd_stage2.py +++ b/catalyst/targets/livecd_stage2.py @@ -16,42 +16,11 @@ class livecd_stage2(StageBase): "boot/kernel", ]) valid_values = required_values | frozenset([ - "livecd/bootargs", - "livecd/cdtar", - "livecd/depclean", - "livecd/empty", - "livecd/fsops", - "livecd/fsscript", - "livecd/fstype", "livecd/gk_mainargs", - "livecd/iso", "livecd/linuxrc", - "livecd/modblacklist", - "livecd/motd", - "livecd/overlay", - "livecd/rcadd", - "livecd/rcdel", - "livecd/readme", - "livecd/rm", - "livecd/root_overlay", - "livecd/type", - "livecd/unmerge", - "livecd/users", - "livecd/verify", - "livecd/volid", - "livecd/xdm", - "livecd/xinitrc", - "livecd/xsession", "portage_overlay", ]) - def __init__(self, spec, addlargs): - StageBase.__init__(self, spec, addlargs) - if "livecd/type" not in self.settings: - self.settings["livecd/type"] = "generic-livecd" - - file_locate(self.settings, ["cdtar", "controller_file"]) - def set_spec_prefix(self): self.settings["spec_prefix"] = "livecd" @@ -65,26 +34,9 @@ class livecd_stage2(StageBase): super(livecd_stage2, self).set_target_path() clear_dir(self.settings['target_path']) - def run_local(self): - # what modules do we want to blacklist? - if "livecd/modblacklist" in self.settings: - path = normpath(self.settings["chroot_path"] + - "/etc/modprobe.d/blacklist.conf") - try: - with open(path, "a") as myf: - myf.write("\n#Added by Catalyst:") - # workaround until config.py is using configparser - if isinstance(self.settings["livecd/modblacklist"], str): - self.settings["livecd/modblacklist"] = self.settings[ - "livecd/modblacklist"].split() - for x in self.settings["livecd/modblacklist"]: - myf.write("\nblacklist "+x) - except: - self.unbind() - raise CatalystError("Couldn't open " + - self.settings["chroot_path"] + - "/etc/modprobe.d/blacklist.conf.", - print_traceback=True) + def set_stage_path(self): + self.set_target_path() + self.settings['stage_path'] = self.settings['target_path'] def set_action_sequence(self): self.settings['action_sequence'] = [ @@ -95,24 +47,11 @@ class livecd_stage2(StageBase): "bind", "chroot_setup", "setup_environment", - "run_local", "build_kernel" ] if "fetch" not in self.settings["options"]: self.settings['action_sequence'] += [ "bootloader", - "preclean", - "livecd_update", - "root_overlay", - "fsscript", - "rcupdate", - "unmerge", "unbind", - "remove", - "empty", - "clean", - "target_setup", - "setup_overlay", - "create_iso", ] - self.settings["action_sequence"].append("clear_autoresume") + self.set_completion_action_sequences() diff --git a/catalyst/targets/livecd_stage3.py b/catalyst/targets/livecd_stage3.py new file mode 100644 index 00000000..a5df72d3 --- /dev/null +++ b/catalyst/targets/livecd_stage3.py @@ -0,0 +1,61 @@ +""" +LiveCD stage3 target +""" +# NOTE: That^^ docstring has influence catalyst-spec(5) man page generation. + +from catalyst.support import normpath +from catalyst.fileops import clear_dir +from catalyst.targets.livecd_stage1 import livecd_stage1 + + +class livecd_stage3(livecd_stage1): + """ + Builder class for LiveCD stage3. + + Wraps any stage into a CD image + + Requires a cached kernel name and cdtar + """ + required_values = frozenset([ + "boot/kernel", + "livecd/cdtar", + ]) + valid_values = required_values | frozenset([ + "livecd/bootargs", + "livecd/empty", + "livecd/fsscript", + "livecd/fstype", + "livecd/iso", + "livecd/rm", + "livecd/root_overlay", + "livecd/type", + "livecd/verify", + "livecd/volid", + ]) + + def set_action_sequence(self): + self.settings['action_sequence'] = [ + "unpack", + "bind", + "setup_environment", + "root_overlay", + "fsscript", + "bootloader", + "unbind", + "remove", + "empty", + "target_setup", + "create_iso", + "clear_autoresume", + "remove_chroot", + ] + + def set_target_path(self): + '''Set the target path for the finished stage. + + This method runs the StageBase.set_target_path method, + and additionally creates a staging directory for assembling + the final components needed to produce the iso image. + ''' + super(livecd_stage1, self).set_target_path() + clear_dir(self.settings['target_path']) diff --git a/targets/livecd-stage1/controller.sh b/targets/livecd-stage1/controller.sh index ae897da9..0d472f1f 100755 --- a/targets/livecd-stage1/controller.sh +++ b/targets/livecd-stage1/controller.sh @@ -12,5 +12,10 @@ case $1 in exec_in_chroot \ ${clst_shdir}/${clst_target}/chroot.sh ;; + + fsscript) + exec_in_chroot ${clst_fsscript} + ;; + esac exit $? diff --git a/targets/livecd-stage3/controller.sh b/targets/livecd-stage3/controller.sh new file mode 100755 index 00000000..632e8ccd --- /dev/null +++ b/targets/livecd-stage3/controller.sh @@ -0,0 +1,51 @@ +#!/bin/bash + +source ${clst_shdir}/support/functions.sh + +## START RUNSCRIPT + +case $1 in + fsscript) + exec_in_chroot ${clst_fsscript} + ;; + + bootloader) + shift + # Here is where we poke in our identifier + touch $1/livecd + + # We create a firmware directory, if necessary + if [ -e ${clst_chroot_path}/lib/firmware.tar.bz2 ] + then + echo "Creating firmware directory in $1" + mkdir -p $1/firmware + # TODO: Unpack firmware into $1/firmware and remove it from the + # chroot so newer livecd-tools/genkernel can find it and unpack it. + fi + + # Move over the readme (if applicable) + if [ -n "${clst_livecd_readme}" ] + then + cp -f ${clst_livecd_readme} $1/README.txt + else + cp -f ${clst_sharedir}/livecd/files/README.txt $1 + fi + + extract_cdtar $1 + + for x in ${clst_boot_kernel}; do + extract_modules ${clst_chroot_path} $x + done + ;; + + target_image_setup) + shift + ${clst_shdir}/support/target_image_setup.sh $1 + ;; + + iso) + shift + ${clst_shdir}/support/create-iso.sh $1 + ;; +esac +exit $? -- 2.26.2