[gentoo-catalyst] [PATCH 1/3] main: group related command line flags

[gentoo-catalyst] [PATCH 1/3] main: group related command line flags

[Mike Frysinger]

This makes the --help output more manageable so people can quickly
scan and skip options that they don't care about.
---
 catalyst/main.py | 48 ++++++++++++++++++++++++++++--------------------
 1 file changed, 28 insertions(+), 20 deletions(-)

diff --git a/catalyst/main.py b/catalyst/main.py
index 03c13c0..e6b6447 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -168,41 +168,49 @@ Using the specfile option (-f, --file) to build a stage target:
 $ catalyst -f stage1-specfile.spec"""
 
 	parser = argparse.ArgumentParser(epilog=epilog, formatter_class=argparse.RawDescriptionHelpFormatter)
-	parser.add_argument('-d', '--debug',
+
+	parser.add_argument('-V', '--version',
+		action='version', version=get_version(),
+		help='display version information')
+
+	group = parser.add_argument_group('Program output options')
+	group.add_argument('-d', '--debug',
 		default=False, action='store_true',
 		help='enable debugging')
-	parser.add_argument('-v', '--verbose',
+	group.add_argument('-v', '--verbose',
 		default=False, action='store_true',
 		help='verbose output')
-	parser.add_argument('-c', '--config',
-		type=FilePath(),
-		help='use specified configuration file')
-	parser.add_argument('-f', '--file',
-		type=FilePath(),
-		help='read specfile')
-	parser.add_argument('-F', '--fetchonly',
-		default=False, action='store_true',
-		help='fetch files only')
-	parser.add_argument('-a', '--clear-autoresume',
+
+	group = parser.add_argument_group('Temporary file management')
+	group.add_argument('-a', '--clear-autoresume',
 		default=False, action='store_true',
 		help='clear autoresume flags')
-	parser.add_argument('-p', '--purge',
+	group.add_argument('-p', '--purge',
 		default=False, action='store_true',
 		help='clear tmp dirs, package cache, autoresume flags')
-	parser.add_argument('-P', '--purgeonly',
+	group.add_argument('-P', '--purgeonly',
 		default=False, action='store_true',
 		help='clear tmp dirs, package cache, autoresume flags and exit')
-	parser.add_argument('-T', '--purgetmponly',
+	group.add_argument('-T', '--purgetmponly',
 		default=False, action='store_true',
 		help='clear tmp dirs and autoresume flags and exit')
-	parser.add_argument('-s', '--snapshot',
+
+	group = parser.add_argument_group('Target/config file management')
+	group.add_argument('-F', '--fetchonly',
+		default=False, action='store_true',
+		help='fetch files only')
+	group.add_argument('-c', '--config',
+		type=FilePath(),
+		help='use specified configuration file')
+	group.add_argument('-f', '--file',
+		type=FilePath(),
+		help='read specfile')
+	group.add_argument('-s', '--snapshot',
 		help='generate a release snapshot')
-	parser.add_argument('-V', '--version',
-		action='version', version=get_version(),
-		help='display version information')
-	parser.add_argument('-C', '--cli',
+	group.add_argument('-C', '--cli',
 		default=[], nargs=argparse.REMAINDER,
 		help='catalyst commandline (MUST BE LAST OPTION)')
+
 	return parser
 
 
-- 
2.5.2

[gentoo-catalyst] [PATCH 2/3] log: new logging module to standardize catalyst output

[Mike Frysinger]

This has everything you could ever want:
- control where output is sent (stdout or a file)
- control over log level
- automatic exit when CRITICAL is used
- automatic colorization of critical/error/warning messages
- explicit control over use of colors
- automatic handling of embedded newlines
- standardized output format
- all logging routed through a single logger (not the root)
- additional log level between warning & info: notice

This just lays the groundwork -- no code is actually converted over
to using this.  That will be done in follow up commit(s).

Note: The default output level has changed from "info" to "notice".
That means the default display won't spam w/irrelevant details.

Example output (only the main.py module is converted):
$ ./bin/wrapper.py -s test
09 Oct 2015 01:34:56 EDT: NOTICE  : Using default Catalyst configuration file: /etc/catalyst/catalyst.conf
ContentsMap: __init__(), search_order = ['pixz', 'lbzip2', 'isoinfo_l', 'squashfs', 'gzip', 'xz', 'bzip2', 'tar', 'isoinfo_f']
Creating Portage tree snapshot test from /usr/portage...
^C

$ ./bin/wrapper.py -s test --debug
09 Oct 2015 01:35:43 EDT: INFO    : main.py:version: Catalyst git
09 Oct 2015 01:35:43 EDT: INFO    : main.py:version: vcs version 4440e8908c677d8764e29b0f127e2dd6c02b7621, date Fri Oct 9 01:28:19 2015 -0400
09 Oct 2015 01:35:43 EDT: INFO    : main.py:version: Copyright 2003-2015 Gentoo Foundation
09 Oct 2015 01:35:43 EDT: INFO    : main.py:version: Copyright 2008-2012 various authors
09 Oct 2015 01:35:43 EDT: INFO    : main.py:version: Distributed under the GNU General Public License version 2.1
09 Oct 2015 01:35:43 EDT: NOTICE  : log.py:notice: Using default Catalyst configuration file: /etc/catalyst/catalyst.conf
09 Oct 2015 01:35:43 EDT: INFO    : main.py:parse_config: Snapshot cache support enabled.
09 Oct 2015 01:35:43 EDT: INFO    : main.py:parse_config: Kernel cache support enabled.
09 Oct 2015 01:35:43 EDT: INFO    : main.py:parse_config: Autoresuming support enabled.
09 Oct 2015 01:35:43 EDT: INFO    : main.py:parse_config: Package cache support enabled.
09 Oct 2015 01:35:43 EDT: INFO    : main.py:parse_config: Seed cache support enabled.
09 Oct 2015 01:35:43 EDT: INFO    : main.py:parse_config: Envscript support enabled.
09 Oct 2015 01:35:43 EDT: DEBUG   : main.py:main: conf_values[options] = set(['snapcache', 'kerncache', 'autoresume', 'pkgcache', 'bindist', 'seedcache'])
^C

$ ./bin/wrapper.py -s test -C /asdf
09 Oct 2015 01:36:59 EDT: NOTICE  : Using default Catalyst configuration file: /etc/catalyst/catalyst.conf
ContentsMap: __init__(), search_order = ['pixz', 'lbzip2', 'isoinfo_l', 'squashfs', 'gzip', 'xz', 'bzip2', 'tar', 'isoinfo_f']

!!! catalyst: Syntax error: 0

09 Oct 2015 01:36:59 EDT: CRITICAL: Could not parse commandline
---
 catalyst/log.py  | 130 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 catalyst/main.py |  30 ++++++++++++-
 2 files changed, 158 insertions(+), 2 deletions(-)
 create mode 100644 catalyst/log.py

diff --git a/catalyst/log.py b/catalyst/log.py
new file mode 100644
index 0000000..bf39116
--- /dev/null
+++ b/catalyst/log.py
@@ -0,0 +1,130 @@
+# Copyright 2003-2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+"""Logging related code
+
+This largely exposes the same interface as the logging module except we add
+another level "notice" between warning & info, and all output goes through
+the "catalyst" logger.
+"""
+
+from __future__ import print_function
+
+import logging
+import logging.handlers
+import os
+import sys
+import time
+
+
+class CatalystLogger(logging.Logger):
+	"""Override the _log member to autosplit on new lines"""
+
+	def _log(self, level, msg, args, **kwargs):
+		"""If given a multiline message, split it"""
+		# We have to interpolate it first in case they spread things out
+		# over multiple lines like: Bad Thing:\n%s\nGoodbye!
+		msg %= args
+		for line in msg.splitlines():
+			super(CatalystLogger, self)._log(level, line, (), **kwargs)
+
+
+# The logger that all output should go through.
+# This is ugly because we want to not perturb the logging module state.
+_klass = logging.getLoggerClass()
+logging.setLoggerClass(CatalystLogger)
+logger = logging.getLogger('catalyst')
+logging.setLoggerClass(_klass)
+del _klass
+
+
+# Set the notice level between warning and info.
+NOTICE = (logging.WARNING + logging.INFO) / 2
+logging.addLevelName(NOTICE, 'NOTICE')
+
+
+# The API we expose to consumers.
+def notice(msg, *args, **kwargs):
+	"""Log a notice message"""
+	logger.log(NOTICE, msg, *args, **kwargs)
+
+def critical(msg, *args, **kwargs):
+	"""Log a critical message and then exit"""
+	status = kwargs.pop('status', 1)
+	logger.critical(msg, *args, **kwargs)
+	sys.exit(status)
+
+error = logger.error
+warning = logger.warning
+info = logger.info
+debug = logger.debug
+
+
+class CatalystFormatter(logging.Formatter):
+	"""Mark bad messages with colors automatically"""
+
+	_COLORS = {
+		'CRITICAL':	'\033[1;35m',
+		'ERROR':	'\033[1;31m',
+		'WARNING':	'\033[1;33m',
+		'DEBUG':	'\033[1;34m',
+	}
+	_NORMAL = '\033[0m'
+
+	@staticmethod
+	def detect_color():
+		"""Figure out whether the runtime env wants color"""
+		if 'NOCOLOR' is os.environ:
+			return False
+		return os.isatty(sys.stdout.fileno())
+
+	def __init__(self, *args, **kwargs):
+		"""Initialize"""
+		color = kwargs.pop('color', None)
+		if color is None:
+			color = self.detect_color()
+		if not color:
+			self._COLORS = {}
+
+		super(CatalystFormatter, self).__init__(*args, **kwargs)
+
+	def format(self, record, **kwargs):
+		"""Format the |record| with our color settings"""
+		#print(dir(record))
+		#print(record.getMessage())
+		msg = super(CatalystFormatter, self).format(record, **kwargs)
+		#print('{', msg, '}')
+		color = self._COLORS.get(record.levelname)
+		if color:
+			return color + msg + self._NORMAL
+		else:
+			return msg
+
+
+def setup_logging(level, output=None, debug=False, color=None):
+	"""Initialize the logging module using the |level| level"""
+	# The incoming level will be things like "info", but setLevel wants
+	# the numeric constant.  Convert it here.
+	level = logging.getLevelName(level.upper())
+
+	# The good stuff.
+	fmt = '%(asctime)s: %(levelname)-8s: '
+	if debug:
+		fmt += '%(filename)s:%(funcName)s: '
+	fmt += '%(message)s'
+
+	# Figure out where to send the log output.
+	if output is None:
+		handler = logging.StreamHandler(stream=sys.stdout)
+	else:
+		handler = logging.FileHandler(output)
+
+	# Use a date format that is readable by humans & machines.
+	# Think e-mail/RFC 2822: 05 Oct 2013 18:58:50 EST
+	tzname = time.strftime('%Z', time.localtime())
+	datefmt = '%d %b %Y %H:%M:%S ' + tzname
+	formatter = CatalystFormatter(fmt, datefmt, color=color)
+	handler.setFormatter(formatter)
+
+	logger.addHandler(handler)
+	logger.setLevel(level)
diff --git a/catalyst/main.py b/catalyst/main.py
index e6b6447..c9a2219 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -18,6 +18,7 @@ from DeComp.definitions import (COMPRESS_DEFINITIONS, DECOMPRESS_DEFINITIONS,
 	CONTENTS_DEFINITIONS)
 from DeComp.contents import ContentsMap
 
+from catalyst import log
 import catalyst.config
 import catalyst.util
 from catalyst.defaults import confdefaults, option_messages
@@ -176,10 +177,23 @@ $ catalyst -f stage1-specfile.spec"""
 	group = parser.add_argument_group('Program output options')
 	group.add_argument('-d', '--debug',
 		default=False, action='store_true',
-		help='enable debugging')
+		help='enable debugging (and default --log-level debug)')
 	group.add_argument('-v', '--verbose',
 		default=False, action='store_true',
-		help='verbose output')
+		help='verbose output (and default --log-level info)')
+	group.add_argument('--log-level',
+		default=None,
+		choices=('critical', 'error', 'warning', 'notice', 'info', 'debug'),
+		help='set verbosity of output')
+	group.add_argument('--log-file',
+		type=FilePath(exists=False),
+		help='write all output to this file (instead of stdout)')
+	group.add_argument('--color',
+		default=None, action='store_true',
+		help='colorize output all the time (default: detect)')
+	group.add_argument('--nocolor',
+		dest='color', action='store_false',
+		help='never colorize output all the time (default: detect)')
 
 	group = parser.add_argument_group('Temporary file management')
 	group.add_argument('-a', '--clear-autoresume',
@@ -218,6 +232,18 @@ def main():
 	parser = get_parser()
 	opts = parser.parse_args(sys.argv[1:])
 
+	# Initialize the logger before anything else.
+	log_level = opts.log_level
+	if log_level is None:
+		if opts.debug:
+			log_level = 'debug'
+		elif opts.verbose:
+			log_level = 'info'
+		else:
+			log_level = 'notice'
+	log.setup_logging(log_level, output=opts.log_file, debug=opts.debug,
+		color=opts.color)
+
 	# Parse the command line options.
 	myconfig = opts.config
 	myspecfile = opts.file
-- 
2.5.2

[gentoo-catalyst] [PATCH 3/3] main: convert to new logging module

[Mike Frysinger]

---
 catalyst/main.py | 104 +++++++++++++++++++++++--------------------------------
 1 file changed, 43 insertions(+), 61 deletions(-)

diff --git a/catalyst/main.py b/catalyst/main.py
index c9a2219..bc77c59 100644
--- a/catalyst/main.py
+++ b/catalyst/main.py
@@ -7,6 +7,7 @@
 # $Id$
 
 import argparse
+import datetime
 import os
 import sys
 
@@ -31,36 +32,36 @@ conf_values={}
 
 
 def version():
-	print get_version()
-	print "Copyright 2003-2008 Gentoo Foundation"
-	print "Copyright 2008-2012 various authors"
-	print "Distributed under the GNU General Public License version 2.1\n"
+	log.info(get_version())
+	log.info('Copyright 2003-%s Gentoo Foundation', datetime.datetime.now().year)
+	log.info('Copyright 2008-2012 various authors')
+	log.info('Distributed under the GNU General Public License version 2.1')
 
 def parse_config(myconfig):
 	# search a couple of different areas for the main config file
 	myconf={}
 	config_file=""
+	default_config_file = '/etc/catalyst/catalyst.conf'
 
 	# first, try the one passed (presumably from the cmdline)
 	if myconfig:
 		if os.path.exists(myconfig):
-			print "Using command line specified Catalyst configuration file, "+myconfig
+			log.notice('Using command line specified Catalyst configuration file: %s',
+				myconfig)
 			config_file=myconfig
 
 		else:
-			print "!!! catalyst: Could not use specified configuration file "+\
-				myconfig
-			sys.exit(1)
+			log.critical('Specified configuration file does not exist: %s', myconfig)
 
 	# next, try the default location
-	elif os.path.exists("/etc/catalyst/catalyst.conf"):
-		print "Using default Catalyst configuration file, /etc/catalyst/catalyst.conf"
-		config_file="/etc/catalyst/catalyst.conf"
+	elif os.path.exists(default_config_file):
+		log.notice('Using default Catalyst configuration file: %s',
+			default_config_file)
+		config_file = default_config_file
 
 	# can't find a config file (we are screwed), so bail out
 	else:
-		print "!!! catalyst: Could not find a suitable configuration file"
-		sys.exit(1)
+		log.critical('Could not find a suitable configuration file')
 
 	# now, try and parse the config file "config_file"
 	try:
@@ -69,8 +70,7 @@ def parse_config(myconfig):
 		myconf.update(myconfig.get_values())
 
 	except Exception:
-		print "!!! catalyst: Unable to parse configuration file, "+myconfig
-		sys.exit(1)
+		log.critical('Could not find parse configuration file: %s', myconfig)
 
 	# now, load up the values into conf_values so that we can use them
 	for x in list(confdefaults):
@@ -90,7 +90,7 @@ def parse_config(myconfig):
 	# print out any options messages
 	for opt in conf_values['options']:
 		if opt in option_messages:
-			print option_messages[opt]
+			log.info(option_messages[opt])
 
 	for key in ["digests", "envscript", "var_tmpfs_portage", "port_logdir",
 				"local_overlay"]:
@@ -102,7 +102,7 @@ def parse_config(myconfig):
 		conf_values["contents"] = myconf["contents"].replace("-", '_')
 
 	if "envscript" in myconf:
-		print "Envscript support enabled."
+		log.info('Envscript support enabled.')
 
 	# take care of any variable substitutions that may be left
 	for x in list(conf_values):
@@ -118,11 +118,8 @@ def import_module(target):
 	try:
 		mod_name = "catalyst.targets." + target
 		module = __import__(mod_name, [],[], ["not empty"])
-	except ImportError as e:
-		print "!!! catalyst: Python module import error: %s " % target + \
-			"in catalyst/targets/ ... exiting."
-		print "ERROR was: ", e
-		sys.exit(1)
+	except ImportError:
+		log.critical('Python module import error: %s', target, exc_info=True)
 	return module
 
 
@@ -278,7 +275,7 @@ def main():
 	parse_config(myconfig)
 
 	conf_values["options"].update(options)
-	#print "MAIN: conf_values['options'] =", conf_values["options"]
+	log.debug('conf_values[options] = %s', conf_values['options'])
 
 	# initialize our contents generator
 	contents_map = ContentsMap(CONTENTS_DEFINITIONS)
@@ -308,14 +305,13 @@ def main():
 
 		# First validate all the requested digests are valid keys.
 		if digests - valid_digests:
-			print
-			print "These are not a valid digest entries:"
-			print ', '.join(digests - valid_digests)
-			print "Valid digest entries:"
-			print ', '.join(sorted(valid_digests))
-			print
-			print "Catalyst aborting...."
-			sys.exit(2)
+			log.critical(
+				'These are not valid digest entries:\n'
+				'%s\n'
+				'Valid digest entries:\n'
+				'%s',
+				', '.join(digests - valid_digests),
+				', '.join(sorted(valid_digests)))
 
 		# Then check for any programs that the hash func requires.
 		for digest in digests:
@@ -326,37 +322,28 @@ def main():
 				if skip_missing:
 					digests.remove(digest)
 					continue
-				print
-				print "digest=" + digest
-				print "\tThe " + hash_map.hash_map[digest].cmd + \
-					" binary was not found. It needs to be in your system path"
-				print
-				print "Catalyst aborting...."
-				sys.exit(2)
+				log.critical(
+					'The "%s" binary needed by digest "%s" was not found. '
+					'It needs to be in your system path.',
+					hash_map.hash_map[digest].cmd, digest)
 
 		# Now reload the config with our updated value.
 		conf_values['digests'] = ' '.join(digests)
 
 	if "hash_function" in conf_values:
 		if conf_values["hash_function"] not in HASH_DEFINITIONS:
-			print
-			print conf_values["hash_function"]+\
-				" is not a valid hash_function entry"
-			print "Valid hash_function entries:"
-			print HASH_DEFINITIONS.keys()
-			print
-			print "Catalyst aborting...."
-			sys.exit(2)
+			log.critical(
+				'%s is not a valid hash_function entry\n'
+				'Valid hash_function entries:\n'
+				'%s', HASH_DEFINITIONS.keys())
 		try:
 			process.find_binary(hash_map.hash_map[conf_values["hash_function"]].cmd)
 		except process.CommandNotFound:
-			print
-			print "hash_function="+conf_values["hash_function"]
-			print "\tThe "+hash_map.hash_map[conf_values["hash_function"]].cmd + \
-				" binary was not found. It needs to be in your system path"
-			print
-			print "Catalyst aborting...."
-			sys.exit(2)
+			log.critical(
+				'The "%s" binary needed by hash_function "%s" was not found. '
+				'It needs to be in your system path.',
+				hash_map.hash_map[conf_values['hash_function']].cmd,
+				conf_values['hash_function'])
 
 	addlargs={}
 
@@ -370,25 +357,20 @@ def main():
 			cmdline.parse_lines(mycmdline)
 			addlargs.update(cmdline.get_values())
 		except CatalystError:
-			print "!!! catalyst: Could not parse commandline, exiting."
-			sys.exit(1)
+			log.critical('Could not parse commandline')
 
 	if "target" not in addlargs:
 		raise CatalystError("Required value \"target\" not specified.")
 
 	if os.getuid() != 0:
 		# catalyst cannot be run as a normal user due to chroots, mounts, etc
-		print "!!! catalyst: This script requires root privileges to operate"
-		sys.exit(2)
+		log.critical('This script requires root privileges to operate')
 
 	# everything is setup, so the build is a go
 	try:
 		success = build_target(addlargs)
 	except KeyboardInterrupt:
-		print "\nCatalyst build aborted due to user interrupt ( Ctrl-C )"
-		print
-		print "Catalyst aborting...."
-		sys.exit(2)
+		log.critical('Catalyst build aborted due to user interrupt (Ctrl-C)')
 	if not success:
 		sys.exit(2)
 	sys.exit(0)
-- 
2.5.2

Re: [gentoo-catalyst] [PATCH 1/3] main: group related command line flags

[Brian Dolbec]

On Fri,  9 Oct 2015 01:57:26 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> This makes the --help output more manageable so people can quickly
> scan and skip options that they don't care about.
> ---
>  catalyst/main.py | 48
> ++++++++++++++++++++++++++++-------------------- 1 file changed, 28
> insertions(+), 20 deletions(-)
> 

Looks good

-- 
Brian Dolbec <dolsen>

Re: [gentoo-catalyst] [PATCH 2/3] log: new logging module to standardize catalyst output

[Brian Dolbec]

On Fri,  9 Oct 2015 01:57:27 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> This has everything you could ever want:
> - control where output is sent (stdout or a file)
> - control over log level
> - automatic exit when CRITICAL is used
> - automatic colorization of critical/error/warning messages
> - explicit control over use of colors
> - automatic handling of embedded newlines
> - standardized output format
> - all logging routed through a single logger (not the root)
> - additional log level between warning & info: notice
> 
> This just lays the groundwork -- no code is actually converted over
> to using this.  That will be done in follow up commit(s).
> 
> Note: The default output level has changed from "info" to "notice".
> That means the default display won't spam w/irrelevant details.
> 
> Example output (only the main.py module is converted):
> $ ./bin/wrapper.py -s test
> 09 Oct 2015 01:34:56 EDT: NOTICE  : Using default Catalyst
> configuration file: /etc/catalyst/catalyst.conf ContentsMap:
> __init__(), search_order = ['pixz', 'lbzip2', 'isoinfo_l',
> 'squashfs', 'gzip', 'xz', 'bzip2', 'tar', 'isoinfo_f'] Creating
> Portage tree snapshot test from /usr/portage... ^C
> 
> $ ./bin/wrapper.py -s test --debug
> 09 Oct 2015 01:35:43 EDT: INFO    : main.py:version: Catalyst git
> 09 Oct 2015 01:35:43 EDT: INFO    : main.py:version: vcs version
> 4440e8908c677d8764e29b0f127e2dd6c02b7621, date Fri Oct 9 01:28:19
> 2015 -0400 09 Oct 2015 01:35:43 EDT: INFO    : main.py:version:
> Copyright 2003-2015 Gentoo Foundation 09 Oct 2015 01:35:43 EDT:
> INFO    : main.py:version: Copyright 2008-2012 various authors 09 Oct
> 2015 01:35:43 EDT: INFO    : main.py:version: Distributed under the
> GNU General Public License version 2.1 09 Oct 2015 01:35:43 EDT:
> NOTICE  : log.py:notice: Using default Catalyst configuration
> file: /etc/catalyst/catalyst.conf 09 Oct 2015 01:35:43 EDT: INFO    :
> main.py:parse_config: Snapshot cache support enabled. 09 Oct 2015
> 01:35:43 EDT: INFO    : main.py:parse_config: Kernel cache support
> enabled. 09 Oct 2015 01:35:43 EDT: INFO    : main.py:parse_config:
> Autoresuming support enabled. 09 Oct 2015 01:35:43 EDT: INFO    :
> main.py:parse_config: Package cache support enabled. 09 Oct 2015
> 01:35:43 EDT: INFO    : main.py:parse_config: Seed cache support
> enabled. 09 Oct 2015 01:35:43 EDT: INFO    : main.py:parse_config:
> Envscript support enabled. 09 Oct 2015 01:35:43 EDT: DEBUG   :
> main.py:main: conf_values[options] = set(['snapcache', 'kerncache',
> 'autoresume', 'pkgcache', 'bindist', 'seedcache']) ^C
> 
> $ ./bin/wrapper.py -s test -C /asdf
> 09 Oct 2015 01:36:59 EDT: NOTICE  : Using default Catalyst
> configuration file: /etc/catalyst/catalyst.conf ContentsMap:
> __init__(), search_order = ['pixz', 'lbzip2', 'isoinfo_l',
> 'squashfs', 'gzip', 'xz', 'bzip2', 'tar', 'isoinfo_f']
> 
> !!! catalyst: Syntax error: 0
> 
> 09 Oct 2015 01:36:59 EDT: CRITICAL: Could not parse commandline
> ---
>  catalyst/log.py  | 130
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++
> catalyst/main.py |  30 ++++++++++++- 2 files changed, 158
> insertions(+), 2 deletions(-) create mode 100644 catalyst/log.py
> 
> diff --git a/catalyst/log.py b/catalyst/log.py
> new file mode 100644
> index 0000000..bf39116
> --- /dev/null
> +++ b/catalyst/log.py
> @@ -0,0 +1,130 @@
> +# Copyright 2003-2015 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +"""Logging related code
> +
> +This largely exposes the same interface as the logging module except
> we add +another level "notice" between warning & info, and all output
> goes through +the "catalyst" logger.
> +"""
> +
> +from __future__ import print_function
> +
> +import logging
> +import logging.handlers
> +import os
> +import sys
> +import time
> +
> +
> +class CatalystLogger(logging.Logger):
> +	"""Override the _log member to autosplit on new lines"""
> +
> +	def _log(self, level, msg, args, **kwargs):
> +		"""If given a multiline message, split it"""
> +		# We have to interpolate it first in case they
> spread things out
> +		# over multiple lines like: Bad Thing:\n%s\nGoodbye!
> +		msg %= args
> +		for line in msg.splitlines():
> +			super(CatalystLogger, self)._log(level,
> line, (), **kwargs) +
> +
> +# The logger that all output should go through.
> +# This is ugly because we want to not perturb the logging module
> state. +_klass = logging.getLoggerClass()
> +logging.setLoggerClass(CatalystLogger)
> +logger = logging.getLogger('catalyst')
> +logging.setLoggerClass(_klass)
> +del _klass
> +
> +
> +# Set the notice level between warning and info.
> +NOTICE = (logging.WARNING + logging.INFO) / 2
> +logging.addLevelName(NOTICE, 'NOTICE')
> +
> +
> +# The API we expose to consumers.
> +def notice(msg, *args, **kwargs):
> +	"""Log a notice message"""
> +	logger.log(NOTICE, msg, *args, **kwargs)
> +
> +def critical(msg, *args, **kwargs):
> +	"""Log a critical message and then exit"""
> +	status = kwargs.pop('status', 1)
> +	logger.critical(msg, *args, **kwargs)
> +	sys.exit(status)
> +
> +error = logger.error
> +warning = logger.warning
> +info = logger.info
> +debug = logger.debug
> +
> +
> +class CatalystFormatter(logging.Formatter):
> +	"""Mark bad messages with colors automatically"""
> +
> +	_COLORS = {
> +		'CRITICAL':	'\033[1;35m',
> +		'ERROR':	'\033[1;31m',
> +		'WARNING':	'\033[1;33m',
> +		'DEBUG':	'\033[1;34m',
> +	}
> +	_NORMAL = '\033[0m'
> +
> +	@staticmethod
> +	def detect_color():
> +		"""Figure out whether the runtime env wants color"""
> +		if 'NOCOLOR' is os.environ:
> +			return False
> +		return os.isatty(sys.stdout.fileno())
> +
> +	def __init__(self, *args, **kwargs):
> +		"""Initialize"""
> +		color = kwargs.pop('color', None)
> +		if color is None:
> +			color = self.detect_color()
> +		if not color:
> +			self._COLORS = {}
> +
> +		super(CatalystFormatter, self).__init__(*args,
> **kwargs) +
> +	def format(self, record, **kwargs):
> +		"""Format the |record| with our color settings"""
> +		#print(dir(record))
> +		#print(record.getMessage())
> +		msg = super(CatalystFormatter, self).format(record,
> **kwargs)
> +		#print('{', msg, '}')
> +		color = self._COLORS.get(record.levelname)
> +		if color:
> +			return color + msg + self._NORMAL
> +		else:
> +			return msg
> +
> +
> +def setup_logging(level, output=None, debug=False, color=None):
> +	"""Initialize the logging module using the |level| level"""
> +	# The incoming level will be things like "info", but
> setLevel wants
> +	# the numeric constant.  Convert it here.
> +	level = logging.getLevelName(level.upper())
> +
> +	# The good stuff.
> +	fmt = '%(asctime)s: %(levelname)-8s: '
> +	if debug:
> +		fmt += '%(filename)s:%(funcName)s: '
> +	fmt += '%(message)s'
> +
> +	# Figure out where to send the log output.
> +	if output is None:
> +		handler = logging.StreamHandler(stream=sys.stdout)
> +	else:
> +		handler = logging.FileHandler(output)
> +
> +	# Use a date format that is readable by humans & machines.
> +	# Think e-mail/RFC 2822: 05 Oct 2013 18:58:50 EST
> +	tzname = time.strftime('%Z', time.localtime())
> +	datefmt = '%d %b %Y %H:%M:%S ' + tzname
> +	formatter = CatalystFormatter(fmt, datefmt, color=color)
> +	handler.setFormatter(formatter)
> +
> +	logger.addHandler(handler)
> +	logger.setLevel(level)
> diff --git a/catalyst/main.py b/catalyst/main.py
> index e6b6447..c9a2219 100644
> --- a/catalyst/main.py
> +++ b/catalyst/main.py
> @@ -18,6 +18,7 @@ from DeComp.definitions import
> (COMPRESS_DEFINITIONS, DECOMPRESS_DEFINITIONS, CONTENTS_DEFINITIONS)
>  from DeComp.contents import ContentsMap
>  
> +from catalyst import log
>  import catalyst.config
>  import catalyst.util
>  from catalyst.defaults import confdefaults, option_messages
> @@ -176,10 +177,23 @@ $ catalyst -f stage1-specfile.spec"""
>  	group = parser.add_argument_group('Program output options')
>  	group.add_argument('-d', '--debug',
>  		default=False, action='store_true',
> -		help='enable debugging')
> +		help='enable debugging (and default --log-level
> debug)') group.add_argument('-v', '--verbose',
>  		default=False, action='store_true',
> -		help='verbose output')
> +		help='verbose output (and default --log-level info)')
> +	group.add_argument('--log-level',
> +		default=None,
> +		choices=('critical', 'error', 'warning', 'notice',
> 'info', 'debug'),
> +		help='set verbosity of output')
> +	group.add_argument('--log-file',
> +		type=FilePath(exists=False),
> +		help='write all output to this file (instead of
> stdout)')
> +	group.add_argument('--color',
> +		default=None, action='store_true',
> +		help='colorize output all the time (default:
> detect)')
> +	group.add_argument('--nocolor',
> +		dest='color', action='store_false',
> +		help='never colorize output all the time (default:
> detect)') 
>  	group = parser.add_argument_group('Temporary file
> management') group.add_argument('-a', '--clear-autoresume',
> @@ -218,6 +232,18 @@ def main():
>  	parser = get_parser()
>  	opts = parser.parse_args(sys.argv[1:])
>  
> +	# Initialize the logger before anything else.
> +	log_level = opts.log_level
> +	if log_level is None:
> +		if opts.debug:
> +			log_level = 'debug'
> +		elif opts.verbose:
> +			log_level = 'info'
> +		else:
> +			log_level = 'notice'
> +	log.setup_logging(log_level, output=opts.log_file,
> debug=opts.debug,
> +		color=opts.color)
> +
>  	# Parse the command line options.
>  	myconfig = opts.config
>  	myspecfile = opts.file

Very nice. :)   I learned a bit in here ;)

In gkeys I had it log to a file always, but didn't add color to the
console output.

-- 
Brian Dolbec <dolsen>

Re: [gentoo-catalyst] [PATCH 3/3] main: convert to new logging module

[Brian Dolbec]

On Fri,  9 Oct 2015 01:57:28 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> ---
>  catalyst/main.py | 104
> +++++++++++++++++++++++-------------------------------- 1 file
> changed, 43 insertions(+), 61 deletions(-)
> 
> diff --git a/catalyst/main.py b/catalyst/main.py
> index c9a2219..bc77c59 100644
> --- a/catalyst/main.py
> +++ b/catalyst/main.py
> @@ -7,6 +7,7 @@
>  # $Id$
>  
>  import argparse
> +import datetime
>  import os
>  import sys
>  
> @@ -31,36 +32,36 @@ conf_values={}
>  
>  
>  def version():
> -	print get_version()
> -	print "Copyright 2003-2008 Gentoo Foundation"
> -	print "Copyright 2008-2012 various authors"
> -	print "Distributed under the GNU General Public License
> version 2.1\n"
> +	log.info(get_version())
> +	log.info('Copyright 2003-%s Gentoo Foundation',
> datetime.datetime.now().year)
> +	log.info('Copyright 2008-2012 various authors')
> +	log.info('Distributed under the GNU General Public License
> version 2.1') 
>  def parse_config(myconfig):
>  	# search a couple of different areas for the main config file
>  	myconf={}
>  	config_file=""
> +	default_config_file = '/etc/catalyst/catalyst.conf'
>  
>  	# first, try the one passed (presumably from the cmdline)
>  	if myconfig:
>  		if os.path.exists(myconfig):
> -			print "Using command line specified Catalyst
> configuration file, "+myconfig
> +			log.notice('Using command line specified
> Catalyst configuration file: %s',
> +				myconfig)
>  			config_file=myconfig
>  
>  		else:
> -			print "!!! catalyst: Could not use specified
> configuration file "+\
> -				myconfig
> -			sys.exit(1)
> +			log.critical('Specified configuration file
> does not exist: %s', myconfig) 
>  	# next, try the default location
> -	elif os.path.exists("/etc/catalyst/catalyst.conf"):
> -		print "Using default Catalyst configuration
> file, /etc/catalyst/catalyst.conf"
> -		config_file="/etc/catalyst/catalyst.conf"
> +	elif os.path.exists(default_config_file):
> +		log.notice('Using default Catalyst configuration
> file: %s',
> +			default_config_file)
> +		config_file = default_config_file
>  
>  	# can't find a config file (we are screwed), so bail out
>  	else:
> -		print "!!! catalyst: Could not find a suitable
> configuration file"
> -		sys.exit(1)
> +		log.critical('Could not find a suitable
> configuration file') 
>  	# now, try and parse the config file "config_file"
>  	try:
> @@ -69,8 +70,7 @@ def parse_config(myconfig):
>  		myconf.update(myconfig.get_values())
>  
>  	except Exception:
> -		print "!!! catalyst: Unable to parse configuration
> file, "+myconfig
> -		sys.exit(1)
> +		log.critical('Could not find parse configuration
> file: %s', myconfig) 
>  	# now, load up the values into conf_values so that we can
> use them for x in list(confdefaults):
> @@ -90,7 +90,7 @@ def parse_config(myconfig):
>  	# print out any options messages
>  	for opt in conf_values['options']:
>  		if opt in option_messages:
> -			print option_messages[opt]
> +			log.info(option_messages[opt])
>  
>  	for key in ["digests", "envscript", "var_tmpfs_portage",
> "port_logdir", "local_overlay"]:
> @@ -102,7 +102,7 @@ def parse_config(myconfig):
>  		conf_values["contents"] =
> myconf["contents"].replace("-", '_') 
>  	if "envscript" in myconf:
> -		print "Envscript support enabled."
> +		log.info('Envscript support enabled.')
>  
>  	# take care of any variable substitutions that may be left
>  	for x in list(conf_values):
> @@ -118,11 +118,8 @@ def import_module(target):
>  	try:
>  		mod_name = "catalyst.targets." + target
>  		module = __import__(mod_name, [],[], ["not empty"])
> -	except ImportError as e:
> -		print "!!! catalyst: Python module import error: %s
> " % target + \
> -			"in catalyst/targets/ ... exiting."
> -		print "ERROR was: ", e
> -		sys.exit(1)
> +	except ImportError:
> +		log.critical('Python module import error: %s',
> target, exc_info=True) return module
>  
>  
> @@ -278,7 +275,7 @@ def main():
>  	parse_config(myconfig)
>  
>  	conf_values["options"].update(options)
> -	#print "MAIN: conf_values['options'] =",
> conf_values["options"]
> +	log.debug('conf_values[options] = %s',
> conf_values['options']) 
>  	# initialize our contents generator
>  	contents_map = ContentsMap(CONTENTS_DEFINITIONS)
> @@ -308,14 +305,13 @@ def main():
>  
>  		# First validate all the requested digests are valid
> keys. if digests - valid_digests:
> -			print
> -			print "These are not a valid digest entries:"
> -			print ', '.join(digests - valid_digests)
> -			print "Valid digest entries:"
> -			print ', '.join(sorted(valid_digests))
> -			print
> -			print "Catalyst aborting...."
> -			sys.exit(2)
> +			log.critical(
> +				'These are not valid digest
> entries:\n'
> +				'%s\n'
> +				'Valid digest entries:\n'
> +				'%s',
> +				', '.join(digests - valid_digests),
> +				', '.join(sorted(valid_digests)))
>  
>  		# Then check for any programs that the hash func
> requires. for digest in digests:
> @@ -326,37 +322,28 @@ def main():
>  				if skip_missing:
>  					digests.remove(digest)
>  					continue
> -				print
> -				print "digest=" + digest
> -				print "\tThe " +
> hash_map.hash_map[digest].cmd + \
> -					" binary was not found. It
> needs to be in your system path"
> -				print
> -				print "Catalyst aborting...."
> -				sys.exit(2)
> +				log.critical(
> +					'The "%s" binary needed by
> digest "%s" was not found. '
> +					'It needs to be in your
> system path.',
> +
> hash_map.hash_map[digest].cmd, digest) 
>  		# Now reload the config with our updated value.
>  		conf_values['digests'] = ' '.join(digests)
>  
>  	if "hash_function" in conf_values:
>  		if conf_values["hash_function"] not in
> HASH_DEFINITIONS:
> -			print
> -			print conf_values["hash_function"]+\
> -				" is not a valid hash_function entry"
> -			print "Valid hash_function entries:"
> -			print HASH_DEFINITIONS.keys()
> -			print
> -			print "Catalyst aborting...."
> -			sys.exit(2)
> +			log.critical(
> +				'%s is not a valid hash_function
> entry\n'
> +				'Valid hash_function entries:\n'
> +				'%s', HASH_DEFINITIONS.keys())
>  		try:
>  			process.find_binary(hash_map.hash_map[conf_values["hash_function"]].cmd)
>  		except process.CommandNotFound:
> -			print
> -			print
> "hash_function="+conf_values["hash_function"]
> -			print "\tThe
> "+hash_map.hash_map[conf_values["hash_function"]].cmd + \
> -				" binary was not found. It needs to
> be in your system path"
> -			print
> -			print "Catalyst aborting...."
> -			sys.exit(2)
> +			log.critical(
> +				'The "%s" binary needed by
> hash_function "%s" was not found. '
> +				'It needs to be in your system
> path.',
> +
> hash_map.hash_map[conf_values['hash_function']].cmd,
> +				conf_values['hash_function'])
>  
>  	addlargs={}
>  
> @@ -370,25 +357,20 @@ def main():
>  			cmdline.parse_lines(mycmdline)
>  			addlargs.update(cmdline.get_values())
>  		except CatalystError:
> -			print "!!! catalyst: Could not parse
> commandline, exiting."
> -			sys.exit(1)
> +			log.critical('Could not parse commandline')
>  
>  	if "target" not in addlargs:
>  		raise CatalystError("Required value \"target\" not
> specified.") 
>  	if os.getuid() != 0:
>  		# catalyst cannot be run as a normal user due to
> chroots, mounts, etc
> -		print "!!! catalyst: This script requires root
> privileges to operate"
> -		sys.exit(2)
> +		log.critical('This script requires root privileges
> to operate') 
>  	# everything is setup, so the build is a go
>  	try:
>  		success = build_target(addlargs)
>  	except KeyboardInterrupt:
> -		print "\nCatalyst build aborted due to user
> interrupt ( Ctrl-C )"
> -		print
> -		print "Catalyst aborting...."
> -		sys.exit(2)
> +		log.critical('Catalyst build aborted due to user
> interrupt (Ctrl-C)') if not success:
>  		sys.exit(2)
>  	sys.exit(0)


Nice :)


-- 
Brian Dolbec <dolsen>