[gentoo-releng] embedded extensions for catalyst

Preamble

  It has become apparent through my development with the
gentoo-embedded project that the project needs a way of dynamically
and predicably producing 'embedded' gentoo images.  At first I took it
upon myself to write my own little program which proved useful, but
not extensible or convenient.
Some time later catalyst was released.  It automatically builds gentoo
stage1, stage2, and stage3 tarballs for gentoo base systems.  With a
bit of modification it looked as though it could be used for building
embedded images.

Short Term Catalyst Requirements

To get 'stage3' target produced with embedded methodologies there are
a few requirements:

    * Use a native stage{2,3} to produce a cross-capable stage{1,2,3}
    * Inform portage of the proper cross-compiler use
    * Merge cross packages into a non-native stageroot
    * Perform various 'house cleaning' tasks on a non-native stage3 image

To get a bit more in-depth:

embedded-stage1:
This image is produced from a native stage{2,3}.  It can include, some
sort of non-native target.  A Non-native target produces binaries that
the native system does not have the capability to.  This could be
something like a cross libc(uClibc-cross) or a architechture cross
gcc(powerpc-cross).

embedded-stage2:
This image is produced from a embedded-stage1 and includes
cross-merged packages.  These packages are merged into a
user-defineable sub-directory of the embedded-stage1 stageroot.  Since
these binaries might not be executable on the current architechture,
they must be merged into this seperate directory.  This also allows
for a native portage, compiler, and system utils(from the
embedded-stage1) to still allow us modify the  embedded-stage2 target.
The tarball produced at the end is the contents of the user-defined
sub-directory into which packages are merged.

embedded-stage3:
This image is produced from a embedded-stage2 and allows
user-definable 'house cleaning' procedures to be performed.  This should
include but is not limited to: unmerging unneeded packages, removing
unneeded directories and files(ex: rm -rf /usr/share/man), arbitrary
shell script/commands executed for final packaged(ex: place into cramfs).

Long Term Catalyst Requirements

In attempting to implement some of the previous requirements myself I
noticed the following improvements on catalyst a much more effective
tool in the future.

  * Allow developers to easily extend catalyst with their own custom
    non-standard stage targets using object-oriented design.
  * Catalyst should be able to load and integrate these standalone plug-in's
    dynamically at run time into itself.
    
This would allow newer projects and end users to create their own
stage targets(like the embedded project) without having to modify the
catalyst codebase.  

There has also been interest in using the native filesystem cross
toolchain to produce these embedded stages rather than having self
contained stages with toolchains inside.  This is most likely possible
but warrents some additional investigation for a method of easily
integrating it.

End Notes

The goal of this document was to layout the requirements of the
gentoo-embedded project to streamline production of embedded images.
Further steps were taken to make future efforts in customizing images
easier for other individuals and projects.  This includes long term
changes to the design of catalyst in order to accomidate modularity.

Ideally these short term goals could be implemented into catalyst
fairly soon as well as putting the long term goals into the catalyst
release schedule.

-------------------

Notes on the attached code:

I attached code that provides basic 'embedded-stage3' capabilities in
catalyst.  I attached two files:

   embedded-catalyst.patch
   embedded.sh

to make the embedded images one must make directory 'embedded' in the
catalyst targets directory, and place the embedded.sh script in there:

	 /usr/local/catalyst/targets/embedded/embedded.sh

then apply the patch to catalyst.  Instead of a 'stage3' target one
can now use 'embedded'.  Be warned it does not check dependancies for
packages, this is so one can only put in what is needed.