On Sat, Dec 14, 2013 at 02:29:41PM -0800, W. Trevor King wrote: > Oops, I meant to include a diff from v1 in the v2 cover letter. Here > it is now: I've also rebased origin/rewrite-on-master onto the top of this series, so we know where we stand: The following changes since commit e7dade2e894ba56ad75d308e3531ac1656282f62: catalyst: cleanup long lines (2013-12-14 14:16:28 -0800) are available in the git repository at: git://tremily.us/catalyst.git rewrite-without-wtk for you to fetch changes up to c021dbf9703d4b00bc609b27fd805284325b0a57: Strip trailing slash from subpath's if they exist when creating filenames. (2013-12-14 14:50:25 -0800) ---------------------------------------------------------------- Brian Dolbec (60): Initial rearrangement of the python directories move catalyst_support, builder, catalyst_lock out of modules, into the catalyst's base namespace rename the modules subpkg to targets, it better reflects what it contains rename files directory to etc to better reflect the directories contents Remove unused urllib import. Remove unused variable new and an undefined variable s. Fix undefined variable: RLIMIT_NOFILE Initial creation of a defaults file and Split up support.py Move confdefaults out of main.py Some options cleanup, unifying their use, reducing redundancy. Move LockInUse from support.py to lock.py, fix bad execption raising, pyflakes cleanup Massive pyflakes import cleanup and broken CatalystError calls. Begin splitting up generic_stage_target into smaller code blocks. some spacing and comment and indent cleanup, etc. remove redundant /bin/bash additions in cmd() calls Move base stage and target files to thier own sub-pkg Update module loading for the new python structure, rename snapshot_target to snapshot fix options being reset by a config file Fix mounts, mountmap hardcoding removal... Use normpath from support chmod +x all sh scripts so they can run from the git checkout Rename all target .py files and classes without _target in them. FIXME! Comment out a small code block causing TypeError. Break out more repeated (path1 + path2)'s... reduce 2 operations into one simpler one Extend ParserBase to do variable substitution. Make shdir a complete path to ease it's use. Add '--- 'to at the start of all echo messages to aid debugging Fix a relative path bug Remove trailing slash for consistency in variables FIXME! Add a forced debug print statement in cmd() for better debug output Commit my testpath file with instructions to run the git checkout code directly without being installed. Add archdir to settings Fix a bug that portage didn't get rebuilt with the build use flag Migrate hardcoded /etc/portage paths mixed spaces/tabs and indent cleanup. update gitignore Fix typo in error message, clean up useless tracebacks Fix broken seed stage update... fix indent. update version to identify the rewite-git code is being run Make the use of preserved_libs optional by setting the options variable. Migrate version to use snakeoil's format_version() to append git commit info. Initial creation of fileops.py Refactor ClearBase code to remove code duplication. Create AutoResume class to handle all file creation, deletion... Fix some paths/normpath usage. Use pjoin to join paths. Use os.path.dirname() instead of splitting and rejoining without the filename. Remove some dead code Fix a long line. Rename local unpack var to _unpack temp set shebang to python2 until py3 compatibilty is done update doc scripts for the new structure Make testpath automagic via dwfeed's tweak fix '-' in stage names for python imports Add set_version command to setup.py. Streamline data_files generation with additional keys main.py, lock.py, fileops.py remove shebangs Fix livecd source and target paths not ending with a slash Add my outgoing directory to ignore Strip trailing slash from subpath's if they exist when creating filenames. Raśl Porcel (1): Add support for m68k W. Trevor King (4): setup.py: Add disutils-based packaging Makefiles: create files directory before populating it Makefile: Set PYTHONPATH=. for make_target_table.py version: Remove shebang (Python modules are not scripts) .gitignore | 10 + MANIFEST.in | 6 + Makefile | 19 +- bin/catalyst | 46 ++ catalyst/__init__.py | 8 + catalyst/arch/__init__.py | 1 + {arch => catalyst/arch}/alpha.py | 5 +- {arch => catalyst/arch}/amd64.py | 2 +- {arch => catalyst/arch}/arm.py | 4 +- {arch => catalyst/arch}/hppa.py | 3 +- {arch => catalyst/arch}/ia64.py | 3 +- catalyst/arch/m68k.py | 21 + {arch => catalyst/arch}/mips.py | 3 +- {arch => catalyst/arch}/powerpc.py | 9 +- {arch => catalyst/arch}/s390.py | 3 +- {arch => catalyst/arch}/sh.py | 3 +- {arch => catalyst/arch}/sparc.py | 9 +- {arch => catalyst/arch}/x86.py | 9 +- catalyst/base/__init__.py | 1 + catalyst/base/clearbase.py | 83 +++ catalyst/base/genbase.py | 58 ++ catalyst/base/resume.py | 166 +++++ .../base/stagebase.py | 784 +++++++++------------ catalyst/base/targetbase.py | 13 + {modules => catalyst}/builder.py | 0 {modules/catalyst => catalyst}/config.py | 19 +- catalyst/contents.py | 72 ++ catalyst/defaults.py | 111 +++ catalyst/fileops.py | 44 ++ catalyst/hash_utils.py | 106 +++ modules/catalyst_lock.py => catalyst/lock.py | 95 ++- catalyst => catalyst/main.py | 258 +++---- catalyst/support.py | 319 +++++++++ catalyst/targets/__init__.py | 1 + .../targets/embedded.py | 16 +- modules/grp_target.py => catalyst/targets/grp.py | 54 +- .../targets/livecd_stage1.py | 44 +- .../targets/livecd_stage2.py | 85 +-- .../targets/netboot.py | 53 +- .../targets/netboot2.py | 82 ++- .../targets/snapshot.py | 32 +- .../stage1_target.py => catalyst/targets/stage1.py | 40 +- .../stage2_target.py => catalyst/targets/stage2.py | 33 +- .../stage3_target.py => catalyst/targets/stage3.py | 17 +- catalyst/targets/stage4.py | 39 + .../targets/tinderbox.py | 28 +- {modules/catalyst => catalyst}/util.py | 0 catalyst/version.py | 59 ++ doc/HOWTO.txt | 10 +- doc/catalyst-spec.5.txt | 4 +- doc/make_subarch_table_guidexml.py | 5 +- doc/make_target_table.py | 13 +- {files => etc}/.gitignore | 0 {files => etc}/catalyst.conf | 22 +- {files => etc}/catalystrc | 0 modules/catalyst/__init__.py | 0 modules/catalyst_support.py | 718 ------------------- modules/generic_target.py | 11 - modules/stage4_target.py | 43 -- setup.py | 123 ++++ targets/embedded/embedded-chroot.sh | 0 targets/embedded/embedded-controller.sh | 22 +- targets/embedded/embedded-fs-runscript.sh | 0 targets/embedded/embedded-preclean-chroot.sh | 0 targets/embedded/unmerge.sh | 0 targets/grp/grp-chroot.sh | 0 targets/grp/grp-controller.sh | 6 +- targets/grp/grp-preclean-chroot.sh | 0 targets/livecd-stage1/livecd-stage1-chroot.sh | 0 targets/livecd-stage1/livecd-stage1-controller.sh | 4 +- .../livecd-stage1/livecd-stage1-preclean-chroot.sh | 0 targets/livecd-stage2/livecd-stage2-controller.sh | 22 +- targets/netboot/netboot-chroot.sh | 0 targets/netboot/netboot-combine.sh | 6 +- targets/netboot/netboot-controller.sh | 24 +- targets/netboot/netboot-image.sh | 0 targets/netboot2/netboot2-controller.sh | 16 +- targets/netboot2/netboot2-copyfile.sh | 0 targets/netboot2/netboot2-pkg.sh | 0 targets/stage1/build.py | 4 +- targets/stage1/stage1-chroot.sh | 25 +- targets/stage1/stage1-controller.sh | 12 +- targets/stage1/stage1-preclean-chroot.sh | 0 targets/stage2/stage2-chroot.sh | 0 targets/stage2/stage2-controller.sh | 6 +- targets/stage2/stage2-preclean-chroot.sh | 0 targets/stage3/stage3-chroot.sh | 0 targets/stage3/stage3-controller.sh | 6 +- targets/stage3/stage3-preclean-chroot.sh | 0 targets/stage4/stage4-chroot.sh | 0 targets/stage4/stage4-controller.sh | 22 +- targets/stage4/stage4-preclean-chroot.sh | 0 targets/support/bootloader-setup.sh | 6 +- targets/support/chroot-functions.sh | 29 +- targets/support/create-iso.sh | 4 +- targets/support/filesystem-functions.sh | 0 targets/support/functions.sh | 18 +- targets/support/kill-chroot-pids.sh | 0 targets/support/kmerge.sh | 20 +- targets/support/livecdfs-update.sh | 12 +- targets/support/mips-arcload_conf.sh | 0 targets/support/netboot2-final.sh | 4 +- targets/support/post-kmerge.sh | 0 targets/support/pre-kmerge.sh | 0 targets/support/rc-update.sh | 0 targets/support/target_image_setup.sh | 4 +- targets/support/unmerge.sh | 0 targets/tinderbox/tinderbox-chroot.sh | 0 targets/tinderbox/tinderbox-controller.sh | 6 +- targets/tinderbox/tinderbox-preclean-chroot.sh | 0 testpath | 18 + 111 files changed, 2277 insertions(+), 1844 deletions(-) create mode 100644 MANIFEST.in create mode 100755 bin/catalyst create mode 100644 catalyst/__init__.py create mode 100644 catalyst/arch/__init__.py rename {arch => catalyst/arch}/alpha.py (98%) rename {arch => catalyst/arch}/amd64.py (99%) rename {arch => catalyst/arch}/arm.py (98%) rename {arch => catalyst/arch}/hppa.py (95%) rename {arch => catalyst/arch}/ia64.py (89%) create mode 100644 catalyst/arch/m68k.py rename {arch => catalyst/arch}/mips.py (99%) rename {arch => catalyst/arch}/powerpc.py (95%) rename {arch => catalyst/arch}/s390.py (95%) rename {arch => catalyst/arch}/sh.py (98%) rename {arch => catalyst/arch}/sparc.py (85%) rename {arch => catalyst/arch}/x86.py (95%) create mode 100644 catalyst/base/__init__.py create mode 100644 catalyst/base/clearbase.py create mode 100644 catalyst/base/genbase.py create mode 100644 catalyst/base/resume.py rename modules/generic_stage_target.py => catalyst/base/stagebase.py (69%) create mode 100644 catalyst/base/targetbase.py rename {modules => catalyst}/builder.py (100%) rename {modules/catalyst => catalyst}/config.py (87%) create mode 100644 catalyst/contents.py create mode 100644 catalyst/defaults.py create mode 100644 catalyst/fileops.py create mode 100644 catalyst/hash_utils.py rename modules/catalyst_lock.py => catalyst/lock.py (87%) rename catalyst => catalyst/main.py (55%) create mode 100644 catalyst/support.py create mode 100644 catalyst/targets/__init__.py rename modules/embedded_target.py => catalyst/targets/embedded.py (85%) rename modules/grp_target.py => catalyst/targets/grp.py (73%) rename modules/livecd_stage1_target.py => catalyst/targets/livecd_stage1.py (71%) rename modules/livecd_stage2_target.py => catalyst/targets/livecd_stage2.py (66%) rename modules/netboot_target.py => catalyst/targets/netboot.py (69%) rename modules/netboot2_target.py => catalyst/targets/netboot2.py (69%) rename modules/snapshot_target.py => catalyst/targets/snapshot.py (79%) rename modules/stage1_target.py => catalyst/targets/stage1.py (75%) rename modules/stage2_target.py => catalyst/targets/stage2.py (70%) rename modules/stage3_target.py => catalyst/targets/stage3.py (65%) create mode 100644 catalyst/targets/stage4.py rename modules/tinderbox_target.py => catalyst/targets/tinderbox.py (61%) rename {modules/catalyst => catalyst}/util.py (100%) create mode 100644 catalyst/version.py rename {files => etc}/.gitignore (100%) rename {files => etc}/catalyst.conf (90%) rename {files => etc}/catalystrc (100%) delete mode 100644 modules/catalyst/__init__.py delete mode 100644 modules/catalyst_support.py delete mode 100644 modules/generic_target.py delete mode 100644 modules/stage4_target.py create mode 100755 setup.py mode change 100644 => 100755 targets/embedded/embedded-chroot.sh mode change 100644 => 100755 targets/embedded/embedded-controller.sh mode change 100644 => 100755 targets/embedded/embedded-fs-runscript.sh mode change 100644 => 100755 targets/embedded/embedded-preclean-chroot.sh mode change 100644 => 100755 targets/embedded/unmerge.sh mode change 100644 => 100755 targets/grp/grp-chroot.sh mode change 100644 => 100755 targets/grp/grp-controller.sh mode change 100644 => 100755 targets/grp/grp-preclean-chroot.sh mode change 100644 => 100755 targets/livecd-stage1/livecd-stage1-chroot.sh mode change 100644 => 100755 targets/livecd-stage1/livecd-stage1-controller.sh mode change 100644 => 100755 targets/livecd-stage1/livecd-stage1-preclean-chroot.sh mode change 100644 => 100755 targets/livecd-stage2/livecd-stage2-controller.sh mode change 100644 => 100755 targets/netboot/netboot-chroot.sh mode change 100644 => 100755 targets/netboot/netboot-combine.sh mode change 100644 => 100755 targets/netboot/netboot-controller.sh mode change 100644 => 100755 targets/netboot/netboot-image.sh mode change 100644 => 100755 targets/netboot2/netboot2-controller.sh mode change 100644 => 100755 targets/netboot2/netboot2-copyfile.sh mode change 100644 => 100755 targets/netboot2/netboot2-pkg.sh mode change 100644 => 100755 targets/stage1/build.py mode change 100644 => 100755 targets/stage1/stage1-chroot.sh mode change 100644 => 100755 targets/stage1/stage1-controller.sh mode change 100644 => 100755 targets/stage1/stage1-preclean-chroot.sh mode change 100644 => 100755 targets/stage2/stage2-chroot.sh mode change 100644 => 100755 targets/stage2/stage2-controller.sh mode change 100644 => 100755 targets/stage2/stage2-preclean-chroot.sh mode change 100644 => 100755 targets/stage3/stage3-chroot.sh mode change 100644 => 100755 targets/stage3/stage3-controller.sh mode change 100644 => 100755 targets/stage3/stage3-preclean-chroot.sh mode change 100644 => 100755 targets/stage4/stage4-chroot.sh mode change 100644 => 100755 targets/stage4/stage4-controller.sh mode change 100644 => 100755 targets/stage4/stage4-preclean-chroot.sh mode change 100644 => 100755 targets/support/bootloader-setup.sh mode change 100644 => 100755 targets/support/chroot-functions.sh mode change 100644 => 100755 targets/support/create-iso.sh mode change 100644 => 100755 targets/support/filesystem-functions.sh mode change 100644 => 100755 targets/support/functions.sh mode change 100644 => 100755 targets/support/kill-chroot-pids.sh mode change 100644 => 100755 targets/support/kmerge.sh mode change 100644 => 100755 targets/support/livecdfs-update.sh mode change 100644 => 100755 targets/support/mips-arcload_conf.sh mode change 100644 => 100755 targets/support/netboot2-final.sh mode change 100644 => 100755 targets/support/post-kmerge.sh mode change 100644 => 100755 targets/support/pre-kmerge.sh mode change 100644 => 100755 targets/support/rc-update.sh mode change 100644 => 100755 targets/support/target_image_setup.sh mode change 100644 => 100755 targets/support/unmerge.sh mode change 100644 => 100755 targets/tinderbox/tinderbox-chroot.sh mode change 100644 => 100755 targets/tinderbox/tinderbox-controller.sh mode change 100644 => 100755 targets/tinderbox/tinderbox-preclean-chroot.sh create mode 100644 testpath Each of my branches (bugs, releng-link, spec-comments, xsession, pkgcache-warning, sshd-tweak) apply on either side of this series without conflicts, after which the diff vs. origin/rewrite-on-master is: diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py sindex 4ace63d..99f4015 100644 --- a/catalyst/base/stagebase.py +++ b/catalyst/base/stagebase.py @@ -198,11 +198,11 @@ class StageBase(TargetBase, ClearBase, GenBase): "portdir":normpath(self.settings["snapshot_cache_path"]+"/" + self.settings["repo_name"]), "distdir":self.settings["distdir"],"port_tmpdir":"tmpfs"} else: - self.mounts = ["proc", "dev", "distdir", "port_tmpdir"] - self.mountmap = {"proc":"/proc", "dev":"/dev", "devpts":"/dev/pts", + self.mounts=["proc","dev", "distdir", "port_tmpdir"] + self.mountmap={"proc":"/proc", "dev":"/dev", "pts":"/dev/pts", "distdir":self.settings["distdir"], "port_tmpdir":"tmpfs"} if os.uname()[0] == "Linux": - self.mounts.append("devpts") + self.mounts.append("pts") self.set_mounts() @@ -222,7 +222,7 @@ class StageBase(TargetBase, ClearBase, GenBase): print "Location of the kerncache is "+\ self.settings["kerncache_path"] self.mounts.append("kerncache") - self.mountmap["kerncache"] = self.settings["kerncache_path"] + self.mountmap["kerncache"]=self.settings["kerncache_path"] if "ccache" in self.settings["options"]: if "CCACHE_DIR" in os.environ: @@ -235,7 +235,7 @@ class StageBase(TargetBase, ClearBase, GenBase): "Compiler cache support can't be enabled (can't find "+\ ccdir+")") self.mounts.append("ccache") - self.mountmap["ccache"] = ccdir + self.mountmap["ccache"]=ccdir """ for the chroot: """ self.env["CCACHE_DIR"] = self.target_mounts["ccache"] @@ -429,7 +429,7 @@ class StageBase(TargetBase, ClearBase, GenBase): def set_snapshot_path(self): self.settings["snapshot_path"]=normpath(self.settings["storedir"]+\ "/snapshots/" + self.settings["snapshot_name"] + - self.settings["snapshot"].rstrip('/') + ".tar.xz") + self.settings["snapshot"].rstrip('/')+".tar.xz") if os.path.exists(self.settings["snapshot_path"]): self.settings["snapshot_path_hash"]=\ @@ -440,7 +440,7 @@ class StageBase(TargetBase, ClearBase, GenBase): else: self.settings["snapshot_path"]=normpath(self.settings["storedir"]+\ "/snapshots/" + self.settings["snapshot_name"] + - self.settings["snapshot"].rstrip('/') + ".tar.bz2") + self.settings["snapshot"].rstrip('/')+".tar.bz2") if os.path.exists(self.settings["snapshot_path"]): self.settings["snapshot_path_hash"]=\ @@ -806,7 +806,7 @@ class StageBase(TargetBase, ClearBase, GenBase): print "Valid snapshot cache, skipping unpack of portage tree..." unpack=False else: - destdir = normpath(self.settings["chroot_path"] + self.settings["portdir"]) + destdir=normpath(self.settings["chroot_path"] + self.settings["portdir"]) cleanup_errmsg="Error removing existing snapshot directory." cleanup_msg=\ "Cleaning up existing portage tree (This can take a long time)..." @@ -930,10 +930,10 @@ class StageBase(TargetBase, ClearBase, GenBase): if os.uname()[0] == "FreeBSD": if src == "/dev": cmd = "mount -t devfs none " + target - retval = os.system(cmd) + retval=os.system(cmd) else: cmd = "mount_nullfs " + src + " " + target - retval = os.system(cmd) + retval=os.system(cmd) else: if src == "tmpfs": if "var_tmpfs_portage" in self.settings: @@ -944,7 +944,7 @@ class StageBase(TargetBase, ClearBase, GenBase): else: cmd = "mount --bind " + src + " " + target #print "bind(); cmd =", cmd - retval = os.system(cmd) + retval=os.system(cmd) if retval!=0: self.unbind() raise CatalystError("Couldn't bind mount " + src + "\n" + cmd) @@ -971,7 +971,7 @@ class StageBase(TargetBase, ClearBase, GenBase): self.kill_chroot_pids() - retval2 = os.system("umount " + target) + retval2=os.system("umount " + target) if retval2!=0: ouch=1 warn("Couldn't umount bind mount: " + target) diff --git a/catalyst/defaults.py b/catalyst/defaults.py index 78ce29f..58c44c8 100644 --- a/catalyst/defaults.py +++ b/catalyst/defaults.py @@ -57,8 +57,7 @@ contents_definitions = { } -confdefaults = { +confdefaults={ "archdir": "%(PythonDir)s/arch", "distdir": "/usr/portage/distfiles", "hash_function": "crc32", @@ -71,11 +70,10 @@ confdefaults = { "port_tmpdir": "/var/tmp/portage", "PythonDir": "./catalyst", "repo_name": "portage", - "sharedir": "/usr/share/catalyst", + "sharedir": "/usr/lib/catalyst", "shdir": "/usr/lib/catalyst/targets/", "snapshot_cache": "/var/tmp/catalyst/snapshot_cache", "snapshot_name": "portage-", - "snapshot_cache": "/var/tmp/catalyst/snapshot_cache", "storedir": "/var/tmp/catalyst", } diff --git a/catalyst/main.py b/catalyst/main.py old mode 100755 new mode 100644 index 9bac6ce..8546810 --- a/catalyst/main.py +++ b/catalyst/main.py @@ -71,7 +71,7 @@ def parse_config(myconfig): # first, try the one passed (presumably from the cmdline) if myconfig: if os.path.exists(myconfig): - print "Using command line specified Catalyst configuration file, " + \ + print "Using command line specified Catalyst configuration file, "+\ myconfig config_file=myconfig diff --git a/catalyst/targets/snapshot.py b/catalyst/targets/snapshot.py index 1887110..98cae14 100644 --- a/catalyst/targets/snapshot.py +++ b/catalyst/targets/snapshot.py @@ -24,7 +24,7 @@ class snapshot(TargetBase, GenBase): #self.settings=myspec self.settings["target_subpath"]="portage" st=self.settings["storedir"] - self.settings["snapshot_path"] = normpath(st + "/snapshots/" + self.settings["snapshot_path"]=normpath(st + "/snapshots/" + self.settings["snapshot_name"] + self.settings["version_stamp"] + ".tar.bz2") self.settings["tmp_path"]=normpath(st+"/tmp/"+self.settings["target_subpath"]) @@ -54,11 +54,11 @@ class snapshot(TargetBase, GenBase): cmd("rsync -a --delete --exclude /packages/ --exclude /distfiles/ " + "--exclude /local/ --exclude CVS/ --exclude .svn --filter=H_**/files/digest-* " + self.settings["portdir"] + "/ " + mytmp + "/%s/" % self.settings["repo_name"], - "Snapshot failure", env=self.env) + "Snapshot failure",env=self.env) print "Compressing Portage snapshot tarball..." cmd("tar -I lbzip2 -cf " + self.settings["snapshot_path"] + " -C " + - mytmp + " " + self.settings["repo_name"], + mytmp + " %s" % self.settings["repo_name"], "Snapshot creation failure",env=self.env) self.gen_contents_file(self.settings["snapshot_path"]) diff --git a/catalyst/targets/tinderbox.py b/catalyst/targets/tinderbox.py index ed1bc95..2cebaa8 100644 --- a/catalyst/targets/tinderbox.py +++ b/catalyst/targets/tinderbox.py @@ -35,12 +35,8 @@ class tinderbox(StageBase): print_traceback=True) def set_cleanables(self): - self.settings['cleanables'] = [ - '/etc/resolv.conf', - '/var/tmp/*', - '/root/*', - self.settings['portdir'], - ] + self.settings["cleanables"]=["/etc/resolv.conf","/var/tmp/*","/root/*", + self.settings['portdir']] def set_action_sequence(self): #Default action sequence for run method If that looks ok, I suggest we drop origin/rewrite-on-master in favor of wking/rewrite-without-wtk. Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy