[gentoo-catalyst] [PATCH] First pass at a catalyst-spec man page.

[W. Trevor King]

I was reading through the example `.spec` files trying to get a handle
on how things worked, and noticed a good deal of duplication
(including a number of duplicated typos).  In order to avoid
repetition, I've started a `catalyst-spec` man page, condensing the
comments from the example spec files.

If the powers that be decide that this is a useful path forward, my
next step would be to strip the repeated comments out of the example
files, leaving only the comment content that goes beyond the general
descriptions in the man page.
---
 Makefile                 |   11 +-
 doc/.gitignore           |    2 +
 doc/catalyst-spec.5.txt  |  506 ++++++++++++++++++++++++++++++++++++++++++++++
 doc/make_target_table.py |   21 ++
 files/.gitignore         |    1 +
 5 files changed, 538 insertions(+), 3 deletions(-)
 create mode 100644 doc/catalyst-spec.5.txt
 create mode 100755 doc/make_target_table.py

diff --git a/Makefile b/Makefile
index e6204c2..0df5d8c 100644
--- a/Makefile
+++ b/Makefile
@@ -2,21 +2,26 @@
 # Licensed under GPL v2 or later
 
 PACKAGE_VERSION = `fgrep '__version__=' catalyst | sed 's|^__version__="\(.*\)"$$|\1|'`
-EXTRA_DIST = files/catalyst.1
-CLEAN_FILES = $(EXTRA_DIST) doc/subarches.generated.txt
+MAN_PAGES = catalyst.1 catalyst-spec.5
+MAN_PAGE_INCLUDES = doc/subarches.generated.txt doc/targets.generated.txt
+EXTRA_DIST = $(MAN_PAGES:%=files/%)
+CLEAN_FILES = $(EXTRA_DIST) $(MAN_PAGE_INCLUDES)
 
 distdir = catalyst-$(PACKAGE_VERSION)
 
 
 all: $(EXTRA_DIST)
 
-files/catalyst.1: doc/catalyst.1.txt doc/subarches.generated.txt doc/asciidoc.conf Makefile catalyst
+$(MAN_PAGES:%=files/%): files/%: doc/%.txt $(MAN_PAGE_INCLUDES) doc/asciidoc.conf Makefile catalyst
 	a2x --conf-file=doc/asciidoc.conf --attribute="catalystversion=$(PACKAGE_VERSION)" \
 		 --format=manpage -D files "$<"
 
 doc/subarches.generated.txt: $(wildcard modules/catalyst/arch/*.py) doc/make_subarch_table_guidexml.py
 	./doc/make_subarch_table_guidexml.py
 
+doc/targets.generated.txt: doc/make_target_table.py $(wildcard modules/catalyst/targets/*.py)
+	"./$<" > "$@"
+
 clean:
 	rm -f $(CLEAN_FILES)
 	find -name '*.pyo' -delete
diff --git a/doc/.gitignore b/doc/.gitignore
index b998e26..2b8a08c 100644
--- a/doc/.gitignore
+++ b/doc/.gitignore
@@ -1,2 +1,4 @@
 subarches.generated.txt
 subarches.generated.xml
+targets.generated.txt
+targets.generated.xml
diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
new file mode 100644
index 0000000..6c8cfa0
--- /dev/null
+++ b/doc/catalyst-spec.5.txt
@@ -0,0 +1,506 @@
+CATALYST-SPEC(5)
+================
+:man source:   catalyst {catalystversion}
+:man manual:   catalyst {catalystversion}
+
+
+NAME
+----
+catalyst-spec - Catalyst build specification files
+
+
+SYNOPSIS
+--------
+*catalyst* ['OPTIONS'] *-f* 'FILE'
+
+
+DESCRIPTION
+-----------
+
+*catalyst(1)* reads the specification file given with *-f* on the
+command line.  The file contains keyword-argument pairs, one per line,
+separated by a colon (`:`).  Subsequent intented lines continue the
+argument.  Lines starting with `#` and empty lines are interpreted as
+comments.  For example:
+
+  # Select the subarch
+  subarch: athalon-xp
+
+  boot/use:
+    -*
+    multicall
+    readline
+    ssl
+
+The possible keywords and their meanings are as follows:
+
+Basic configuration
+~~~~~~~~~~~~~~~~~~~
+
+*subarch*::
+The subarch can be any of the supported catalyst subarches (example:
+`athlon-xp`).  See the *SUPPORTED ARCHITECTURES* section for a list of
+supported subarches.
+
+*version_stamp*::
+The version stamp is an identifier for the build.  It can be anything
+you wish it to be, but it is usually a date (example: `2006.1`).
+
+*target*::
+The target specifies what target we want catalyst to do (example:
+`stage3`).  Possible targets are:
+
+include::targets.generated.txt[tabsize=4]
+
+*rel_type*::
+The `rel_type` defines what kind of build we are doing (example:
+`default`).  This is merely another identifier, but it useful for
+allowing multiple concurrent builds.  Usually, `default` will suffice.
+
+*profile*::
+This is the system profile to be used by catalyst to build this target
+(example: `default/linux/x86/10.0/`).  It is specified as a relative
+path from `profiles` in your portage snapshot
+
+*snapshot*::
+This specifies which snapshot to use for building this target
+(example: `2006.1`).
+
+*source_subpath*::
+This specifies where the seed stage comes from for this target
+(example: `default/stage3-x86-2006.1`).  The path is relative to
+`$storedir/builds`.  The `rel_type` is also used as a path prefix for
+the seed.
+
+*distcc_hosts*::
+These are the hosts used as distcc slaves when distcc is enabled in
+your `catalyst.conf` (example: `127.0.0.1 192.168.0.1`).  It follows
+the same syntax as `distcc-config --set-hosts` and is entirely
+optional.
+
+*portage_confdir*::
+This is an optional directory containing portage configuration files
+(example: `/etc/portage`).  It follows the same syntax as
+`/etc/portage` and should be consistent across all targets to minimize
+problems.
+
+*portage_overlay*::
+This option specifies the location to a portage overlay that you would
+like to use when building this target (example: `/usr/local/portage`).
+
+*pkgcache_path*::
+This allows the optional directory containing the output packages for
+catalyst (example: `/tmp/packages`).  Mainly used as a way for
+different spec files to access the same cache directory.  Default
+behavior is for this location to be autogenerated under `$storedir`
+based on the spec file.
+
+*kerncache_path*::
+This allows the optional directory containing the output packages for
+kernel builds (example: `/tmp/kernel`).  Mainly used as a way for
+different spec files to access the same cache directory.  Default
+behavior is for this location to be autogenerated under `$storedir`
+based on the spec file.
+
+*<target>/type*::
+This option controls quite a bit of catalyst internals and sets up
+several defaults.  Each type behaves slightly differently and is
+explained below.
+  `gentoo-release-minimal`;; This creates an official minimal InstallCD.
+  `gentoo-release-universal`;; This creates an official universal InstallCD.
+  `gentoo-release-livecd`;; This creates an official LiveCD environment.
+  `gentoo-gamecd`;; This creates an official Gentoo GameCD.
+  `generic-livecd`;; This should be used for all non-official media.
+
+This setting is supported by the `livecd` target.
+
+*<target>/builddate*::
+Set the build date of the `<target>` (example: `20060107`).  This
+setting is supported by the `netboot2` target.
+
+*<target>/readme*::
+This is for the README.txt on the root of the CD.  For Gentoo
+releases, we use a default README.txt, and this will be used on your
+CD if you do not provide one yourself.  We do not use this for the
+official releases.  This setting is supported by the `livecd` target.
+
+*grp*::
+Since GRP is capable of building packages/source sets for more than
+one CD, this defines the layout for the directories under
+`$storedir/builds` (example: `src cd2`).
+
+Compilation
+~~~~~~~~~~~
+
+These options are only available when building a stage1 or stage2
+target and are all optional.  These allow for emulating the changes
+possible during a bootstrap.  Some possible uses of these would be
+building embedded stages requiring a different `CHOST` or building a
+stage2 with NPTL support from a stage1 tarball that is built without
+it.  If left blank, then the catalyst defaults from `arch.py` are
+used.
+
+*chost*::
+This option is used to change the CHOST from what is default in the
+profile to whatever you specify (example: `i686-pc-linux-gnu`).  This
+is useful for building NPTL, for example.
+
+*cflags*::
+This option allows you to change the default `CFLAGS` that will be
+used in building this stage (example: `-Os -pipe -fomit-frame-pointer
+-mcpu=i686`).  This really should remain generic, as putting
+optimizations flags here will build a stage1 tarball that is no longer
+generic.
+
+*cxxflags*::
+This is for setting the `CXXFLAGS` (example: `-Os -pipe
+-fomit-frame-pointer -mcpu=i686`).  Generally, this would be set to
+the same as `CFLAGS`.  In fact, it will mirror `CFLAGS` by default.
+
+*ldflags*::
+Setting this option sets `LDFLAGS` in `make.conf` in your stage
+(example: `-Wl,-O1 -Wl,-z,now`).  This would be useful for setting up
+an embedded or hardened system.
+
+Filesystem
+~~~~~~~~~~
+
+*livecd/fstype*::
+The fstype is used to determine what sort of CD we should build.  This
+is used to set the type of loopback filesystem that we will use on our
+CD.  Possible values are as follows:
+  `squashfs`;; This gives the best compression, but requires a kernel patch.
+  `zisofs`;; This uses in-kernel compression and is supported on all platforms.
+  `normal`;; This creates a loop without compression.
+  `noloop`;; This copies the files to the CD directly, without using a
+             loopback.
+
+*livecd/fsops*::
+The fsops are a list of optional parameters that can be passed to the
+tool which will create the filesystem specified in *livecd/fstype*
+(example: `-root-owned`).  It is valid for the following fstypes:
+`squashfs`, `jffs`, `jffs2`, and `cramfs`.
+
+*livecd/iso*::
+This is the full path and filename to the ISO image that the
+livecd-stage2 target will create (example:
+`/tmp/installcd-x86-minimal.iso`).
+
+*livecd/volid*::
+This option sets the volume ID of the CD created (example: `Gentoo
+Linux 2006.1 X86`).
+
+Bootloader
+~~~~~~~~~~
+
+*livecd/cdtar*::
+The cdtar is essentially the bootloader for the CD.  It also holds the
+main configuration for the bootloader.  On x86/amd64, it also can
+include a small memory testing application, called memtest86+
+(example:
+`/usr/lib/catalyst/livecd/cdtar/isolinux-2.13-memtest86+-cdtar.tar.bz2`).
+
+Kernel and boot issues
+~~~~~~~~~~~~~~~~~~~~~~
+
+*<target>/splash_theme*::
+This is where you set the splash theme (example: `livecd-2006.1`).
+This theme must be present in `/etc/splash`, before the kernel has
+completed building.  This setting is supported by the `stage4` and
+`livecd` targets.
+
+*boot/kernel*::
+This option is used to specify the number of kernels to build and also
+the labels that will be used by the CD bootloader to refer to each
+kernel image (example: `gentoo`).
+
+*boot/kernel/<label>/sources*::
+*netboot/kernel/sources*::
+This option tells catalyst which kernel sources to merge for this
+kernel label (example: `gentoo-sources`).  This can use normal portage
+atoms to specify a specific version.  `<label>` should match one of
+the labels given to *boot/kernel*.
+
+*boot/kernel/<label>/config*::
+*netboot/kernel/config*::
+This option is the full path and filename to a kernel `.config` file
+that is used by genkernel to compile the kernel this label applies to.
+`<label>` should match one of the labels given to *boot/kernel*.
+
+*boot/kernel/<label>/gk_kernargs*::
+This option sets genkernel parameters on a per-kernel basis and
+applies only to this kernel label.  This can be used for building
+options into only a single kernel, where compatibility may be an
+issue.  We do not use this on the official release media, but it
+follows the same syntax as *stage4/gk_mainargs*.  `<label>` should
+match one of the labels given to *boot/kernel*.
+
+*<target>/gk_mainargs*::
+This is a set of arguments that will be passed to genkernel for all
+kernels defined in this target (example: `--lvm --dmraid`).  It is
+useful for passing arguments to genkernel that are not otherwise
+available via the stage4-stage2 spec file.  This option is supported
+by the `stage4` and `livecd` targets.
+
+*boot/kernel/<label>/extraversion*::
+This option appends an extension to the name of your kernel, as viewed
+by a `uname -r`.  This also affects any modules built under this
+kernel label.  This is useful for having two kernels using the same
+sources to keep the modules from overwriting each other.  We do not
+use this on the official media.  `<label>` should match one of the
+labels given to *boot/kernel*.
+
+*boot/kernel/<label>/machine_type*::
+This option is only for ppc64 machines (example: `ibm`).  If used it
+will create the `/etc/yaboot.conf` entry used for booting an ibm
+powerpc machine.  `<label>` should match one of the labels given to
+*boot/kernel*.
+
+*boot/kernel/<label>/console*::
+This is only supported on ppc64 currently.  This entry sets up the
+console boot parameters required for sending the output to the
+appropriate console (example: `tty0 ttyS0`).  `<label>` should match
+one of the labels given to *boot/kernel*.
+
+*<target>/modblacklist*::
+This is for blacklisting modules from being hotplugged that are known
+to cause problems (example: `8139cp`).  Putting a module name here
+will keep it from being auto-loaded, even if it is detected by
+hotplug.  This setting is supported by the `stage4` and `livecd`
+targets.
+
+*netboot/kernel/use*::
+*boot/kernel/<label>/use*::
+This option sets the `USE` flags used to build the kernel and also any
+packages which are defined under this kernel label (example: `pcmcia
+usb -X`).  These `USE` flags are additive from the default `USE` for
+the specified profile.  `<label>` should match one of the labels given
+to *boot/kernel*.
+
+*boot/kernel/<label>/packages*::
+This option is for merging kernel-dependent packages and external
+modules that are configured against this kernel label (example:
+`pcmcia-cs speedtouch slmodem`).  `<label>` should match one of the
+labels given to *boot/kernel*.
+
+*livecd/bootargs*::
+This is a set of arguments that get passed to the bootloader for your
+CD (example: `dokeymap`).  It is used on the x86/amd64 release media
+to enable keymap selection.
+
+Netboot
+~~~~~~~
+
+*<target>/busybox_config*::
+The netboot target builds busybox for its root filesystem.  This
+option is where you specify the full path and filename to your busybox
+configuration (example: `/tmp/busybox.config`).  This setting is
+supported by the `netboot` and `netboot2` targets.
+
+*netboot/base_tarball*::
+This is the full path and filename to the tarball to use as the base
+for the netboot image (example:
+`/usr/lib/catalyst/netboot/netboot-base.tar.bz2`).
+
+Runlevels
+~~~~~~~~~
+
+*<target>/linuxrc*::
+This option allows you to specify your own linuxrc script for
+genkernel to use when building your CD.  This is not checked for
+functionality, so it is up to you to debug your own script.  We do not
+use one for the official release media.  This option is supported by
+the `stage4` and `livecd` targets.
+
+*<target>/rcadd*::
+This is for adding init scripts to runlevels.  The syntax for the init
+script is the script name, followed by a pipe, followed by the
+runlevel in which you want the script to run.  It looks like
+`spind|default` and is space delimited.  We do not use this on the
+official media, as catalyst sets up the runlevels correctly for us.
+This setting is supported by the `stage4` and `livecd` targets.
+
+*<target>/rcdel*::
+This is for removing init script from runlevels.  It is executed after
+the defaults shipped with catalyst, so it is possible to remove the
+defaults using this option.  It can follow the same syntax as
+*stage4/rcadd*, or you can leave the runlevel off to remove the script
+from any runlevels detected.  We do not use this on the official
+media.  This setting is supported by the `stage4` and `livecd` targets.
+
+Packages
+~~~~~~~~
+
+*<target>/use*::
+This is where you will build packages for updating a `stage3`.  These
+packages can be built with customized `USE` settings (example: `ipv6
+socks5 fbcon ncurses readline ssl`).  The settings here are additive
+to the default `USE` configured by the profile.  Leaving this blank
+will default to the system use flags.  It is very possible to cause
+quite a few problems with these, so be careful with whatever `USE`
+flags you add here.  This is generally used for adding some
+functionality that we do not want on by default for all Gentoo users,
+but that we want on by default in our binaries.  This setting is
+supported by the `stage4`, `netboot2`, `tinderbox`, and `grp` targets.
+
+*<target>/packages*::
+This is the set of packages that we will merge into the stage4 tarball
+(example: `livecd-tools dhcpcd acpid apmd gentoo-sources coldplug
+fxload irssi wpa_supplicant`).  They will be built with the `USE`
+flags configured above.  These packages must not depend on a
+configured kernel.  If the package requires a configured kernel, then
+it will be defined elsewhere.  This setting is supported by the
+`stage4`, `netboot2`, and `tinderbox` targets.
+
+*netboot/packages*::
+These package names are also labels used later when determining what
+files to copy into your netboot image (example: `raidtools
+e2fsprogs`).
+
+*<target>/packages/<label>/files*::
+This is where you tell catalyst which files from each package to copy
+into the netboot image.  `<label>` should match one of the labels
+given to *netboot/packages*.  For example:
+
+  netboot/packages/raidtools/files: /sbin/raidstart /sbin/mkraid
+
+This option is supported by the `netboot` and `netboot2` targets.
+
+*netboot/extra_files*::
+This is a list of any other files, not belonging to the above
+packages, that you would wish to have copied into your netboot image
+(example: `/lib/libresolv.so.2 /lib/libnss_compat.so.2`).
+
+*<target>/unmerge*::
+This is a list of packages that will be unmerged after all the kernels
+have been built (example: `autoconf automake libtool m4 bison`).
+There are no checks on these packages, so be careful what you add
+here.  They can potentially break your target.  This setting is
+supported by the `stage4` and `livecd` targets.
+
+*grp/<label>/type*::
+This tells catalyst what type of GRP set this list of packages will
+create (example: `srcset`).  Valid options here are `srcset` or
+`pkgset` to either download the source, or to build packages,
+respectively.  `<label>` should match one of the labels given to
+*grp*.
+
+*grp/<label>/packages*::
+This is our list of packages that will comprise our package set
+(example: `dante tsocks sys-apps/eject minicom`).  Packages listed for
+a `srcset` label should be used for grabbing things that need a
+compiled kernel to build, or things listed in the Handbook that should
+be available before the first reboot during an install.  Pagekages
+listed for a `pkgset` label will be fetched, compiled, and installed
+in the target.  `<label>` should match one of the labels given to
+*grp*.
+
+Miscellaneous
+~~~~~~~~~~~~~
+
+*<target>/fsscript*::
+A fsscript is simply a shell script that is copied into the chroot of
+the build after the kernel(s) and any external modules have been
+compiled and is executed within the chroot.  It can contain any
+commands that are available via the packages installed by our stages
+or by the packages installed during the stage4-stage1 build.  We do
+not use one for the official release media, so there will not be one
+listed below.  The syntax is simply the full path and filename to the
+shell script that you wish to execute (example:
+`/path/to/fsscript.sh`).  The script is copied into the chroot by
+catalyst automatically.  This setting is supported by the `stage4` and
+`livecd` targets.
+
+*<target>/motd*::
+This is for the message of the day.  It is not required for release
+media, as catalyst builds a default motd when the *<target>/type* is
+set to one of the `gentoo-*` options.  This setting overrides the
+default motd even on official media.  This setting is supported by the
+`stage4` and `livecd` targets.
+
+*<target>/root_overlay*::
+This overlay is dropped onto the filesystem within the loop.  This can
+be used for such things as updating configuration files or adding
+anything else you would want within your target filesystem.  Files
+added here are available when `docache` is used.  We do not use this
+on the official media.  This setting is supported by the `stage4` and
+`livecd` targets.
+
+*livecd/overlay*::
+This overlay is dropped onto the target filesystem and is outside any
+loop which has been configured (example: `/tmp/overlay-minimal`).
+This is typically used for adding the documentation, distfiles,
+snapshots, and stages to the official media.  These files will not be
+available if `docache` is enabled, as they are outside the loop.
+
+*<target>/xinitrc*::
+This is used by catalyst to copy the specified file to
+`/etc/X11/xinit/xinitrc` and is used by the *<target>/type*
+`gentoo-gamecd` and `generic-livecd`.  While the file will still be
+copied for any *<target>/type*, catalyst will only create the
+necessary `/etc/startx` for those types, so X will not be
+automatically started.  This is useful also for setting up X on a CD
+where you do not wish X to start automatically.  We do not use this on
+the release media.  This setting is supported by the `stage4` and
+`livecd` targets.
+
+*livecd/xdm*::
+This is used by catalyst to determine which display manager you wish
+to become the default (example: `gdm`).  This is used on the official
+Gentoo LiveCD and is valid for any `livecd/type`.
+
+*livecd/xsession*::
+This is used by catalyst to determine which X session should be
+started by default by the display manager (example: `gnome`).  This is
+used on the official Gentoo LiveCD and is valid for any livecd/type.
+
+*<target>/users*::
+This option is used to create non-root users on your target.  It takes
+a space separated list of user names.  These users will be added to
+the following groups: `users`, `wheel`, `audio`, `games`, `cdrom`, and
+`usb`.  If this is specified in your spec file, then the first user is
+also the user used to start X.  We do not use this on the release
+media.  This setting is supported by the `stage4` and `livecd` targets.
+
+*<target>/empty*::
+This option is used to empty the directories listed (example:
+`/var/tmp /var/cache /var/db`).  It is useful for getting rid of files
+that don't belong to a particular package, or removing files from a
+package that you wish to keep, but won't need the full functionality.
+This setting is supported by the `stage4` and `livecd` targets.
+
+*<target>/rm**::
+This option tells catalyst to clean specific files from the filesystem
+and is very useful in cleaning up stray files in `/etc` left over
+after *stage4/unmerge* (example: `/lib/*.a /usr/lib/*.a`).  This
+setting is supported by the `stage4` and `livecd` targets.
+
+*gamecd/conf*::
+This option is only used when creating a GameCD.  This specifies the
+file that contains the definitions for `GAME_NAME` and
+`GAME_EXECUTABLE`, which are used by the GameCD scripts to set some
+specific options for the game.  This is not used on the release media.
+
+FILES
+-----
+Example specfiles can be found in '/usr/share/doc/catalyst-{catalystversion}/examples'.
+An example configuration file can be found at '/etc/catalyst/catalyst.conf'.
+
+
+SUPPORTED ARCHITECTURES
+-----------------------
+The following table shows the list of supported architectures as well as
+the list of valid strings for key 'subarch'.
+
+include::subarches.generated.txt[tabsize=4]
+
+
+BUGS
+----
+An up-to-date list of Catalyst bugs can always be found listed on the Gentoo
+Linux bug-tracking system at 'http://bugs.gentoo.org'.
+
+
+SEE ALSO
+--------
+*catalyst(1)*
diff --git a/doc/make_target_table.py b/doc/make_target_table.py
new file mode 100755
index 0000000..1bacbe2
--- /dev/null
+++ b/doc/make_target_table.py
@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+# Copyright (C) 2012 W. Trevor King <wking@××××××.edu>
+# Licensed under GPL v2 or later
+
+# This script should be run from the root of the catalyst source.
+
+import sys as _sys
+
+_sys.path.insert(0, 'modules')  # so we can find the `catalyst` module
+
+import catalyst.target as _catalyst_target
+
+
+if __name__ == '__main__':
+    for module_name,module in sorted(_catalyst_target.get_targets().items()):
+        if hasattr(module, '__target_map'):
+            target_name = module.__target_map.keys()[0]
+            print('`{}`;;'.format(target_name))
+            # Replace blank lines with `+` (asciidoc list item continuation)
+            print(module.__doc__.strip().replace('\n\n', '\n+\n'))
+            print('')
diff --git a/files/.gitignore b/files/.gitignore
index 10acc02..88341e5 100644
--- a/files/.gitignore
+++ b/files/.gitignore
@@ -1 +1,2 @@
 catalyst.1
+catalyst-spec.5
-- 
1.7.3.4