[gentoo-soc] Configuration managment system

[Jonas Bernoulli]

Hello

I would like to implement a configuration management system specially
tailored to Gentoo. Ever since I started using Gentoo I expected this
to be implemented anytime soon as it seams to me like such an obvious
extension of the current config protection. I am still surprised that
this has not happened yet.

Since I have given this a lot of thought over the time I have used
Gentoo but never found the time to actually implement it I would like
to use the opportunity that comes with soc.

While protecting modified config files from being overwritten is an
indispensable feature in itself, it is also somewhat lacking. To
overcome some of the limitations of the simple config protection used
in Gentoo it is advisable to keep the /etc directory under version
control making it possible to recover older versions of config files.

Used like this the repository in which /etc lives mainly functions as
a backup of the files it contains. To any programmer who coded using
some version control system it should be apparent that it isn't the
vcs sole purpose to serve as a backup. Any vcs gives you powerful
tools to branch, merge and compare various states of your code among
other things.

For various reasons "etc-under-version-control" loses much of the
power and/or ease of use that comes with the underlying vcs. Much of
this being due to the fact that the files being kept in the repository
are totally unrelated.

The solution to this obviously is to maintain a separate branch for
each package. And just as obvious one big problem is that the config
files of most applications are mangled into the same location namely
/etc (and elsewhere) making hard to impossible to use normal vc
commands related to branches.

This can be overcome by using a wrapper around some vcs that can deal
with the fact that unrelated files are kept in the same location.

Using git this would look like this at it's core.

  export GIT_DIR=/var/confsink
  export GIT_WORK_TREE=/
  mv $GIT_DIR/index.d/$package $GIT_DIR/index
  git symbolic-ref HEAD refs/heads/live/$package

  git do something

  cp $GIT_DIR/index $GIT_DIR/index.d/$package
  unset GIT_DIR
  unset GIT_WORK_TREE

To those not familiar with git what this basically does is to checkout
the branch containing the live files of $package without removing any
files that are not part of the branch - it's faking a checkout without
actually performing one.  It's a bit more complex than that but this
is what basically has to be done since we can not remove files not
belonging to a package from /etc. We are on a live system after all
and these files might be required while we configure $package.

So one could simply write a shell script that extends the above mockup
to be run before and after work on the config files of $package.
While this would be an improvement over etc-under-vc it is still very
impractial to use.


What is needed is (1) a sophisticated wrapper around git which among
other things (2) replaces etc-update and alike and (3) an extension to
portage that automates the creation and updating of package branches
while emerging packages mainly by (but not limited to ) modifying the
merge step.

Lets concentrate on a simplified version of (3) first:

After installing $package to ${D} and after optionally performing the
test step emerge checks if the package contains any config protected
files.  If it does then it checks out the 'vanilla' branch to ${D} or
creates it if it does not exist yet. Then this is pushed to the
repository ($GIT_DIR) to be found somewhere in /var.  This branch is
not intended to be modified by the user so emerge can safely commit
any changes.

The merge step has to be modified so no ._cfgxxxx_file files are
created. Instead:

After the files are copied to the live system and if there are no
conflicts git can not resolve (there will be less conflicts than
currently) the 'vanilla' branch is merged into the 'live' branch.

If on the other hand automatic merging is not possible for all files
then only the conflict-free files are copied to the live system and
the package is added to a list of packages that have to be later
merged.

The live system now looks exactly the same as without confsink only
that the ._cfgxxxx_file files are missing.

After emerge is done it indicating what packages have to be merged.
Instead of etc-update or alike the confsink command-line client has to
be used to do the merge. (2)

This tool won't be limited to the task of replacing etc-update.  It is
also used to commit changes to configuration files made during daily
use.  It should also support many of the operations that the
underlying git supports while providing something like the above
mockup split in the two obvious parts so that git can be used directly
in hairy cases. This tool should feature an  expert (plumbing) mode as
well as a simple (porcelain) mode suitable for people not willing to
learn about git. Generally the interface will be similar to that of
git.


I have left out many important details in above description but it
should be sufficient to get an understanding of what I plan to
implement and show that I have already given this idea and also it's
implementation a lot thought. If something is missing don't assume I
am not aware of it, but it can't hurt if you point it out anyway. For
example I am well aware that a package/branch has to be in a clean
state before files can be copied to the live system, and that it is
critical when this test is performed.  Or a user might wish to add
additional files to a package.

Not only does above description not contain all relevant details it
also does not list most benefits that confsink offers. For one,
confsink would allow easier sharing of configuration between multiple
machines and even users.


In summary this project involves three tasks:

1. Create the confsink command-line tool as a package aware wrapper
   around git and as a replacement for etc-update.
2. Patch emerge as required by confsink and in part making use of
   the confsink tool.
3. Allow converting a pre-confsink installation to use confsink.

And if there is enough time some extensions could be implemented:

4. Provide easy way to share package configuration.
5. Extend catalyst and/or metro to build a stage5 (sic) which includes
   configuration of selected packages.
6. Emacs interface.
7. Document it so that other people can implement support in other package
   managers not necessarly limited to paludis and pkgcore.

Confsink will be implemented in python and bash, similar to portage.
While I haven't used python much lately that will not be a problem.  I
have done a lot of git surgery lately and enjoy learning about it's
internals so am able to deal with that too. (I am working on a package
manager for Emacs Lisp packages making extensive use of git. And I
have written some tools to import data into git from rsync and files
linked on webpages.) I have also read some of portage's and git's
source code in the past.


  Hopefully I got you interested in this idea.

  Best Regards,

  Jonas