[gentoo-cluster] Adapting java-config for mpi-config

[Donnie Berkholz]

This message about java-config just passed across gentoo-dev. Apparently
it can switch both on a system and a user level, so it may be worth
looking into for adapting to mpi-config.

Thanks,
Donnie

-------- Original Message --------
Subject: [gentoo-dev] Changes to the way Java packages are built
Date: Sun, 18 Jun 2006 10:39:06 -0500
From: Joshua Nichols <nichoj@g.o>
Reply-To: gentoo-dev@l.g.o
To: gentoo-dev@l.g.o

= Background =

As some might have noticed, Java 1.5 has been package.masked for some
time now. There are a number of issues introduced with 1.5 that have
kept it in package.mask. Please see the Java 1.5 FAQ [1] for more details.

[1] http://www.gentoo.org/proj/en/java/tiger-faq.xml

About a year ago, work was begun on improving our part of the build
system (read: Java related eclasses and our java-config tool) in a way
to make it much more flexible in general, but specifically improve it to
get around the known issues. It took about six months to fully develop.
Unfortunately, the new system was not quite a drop-in replacement. So,
it took from then until now to determine how to migrate from the current
system  to the new one in a sane way.


= The Current System =

To give some proper background, it is worth going over the current
system briefly.

== The Java Virtual Machine ==
Each Java Virtual Machine (VM) installs an environment file into
/etc/env.d/java. These files contain information about where JAVA_HOME
is, the PATH to include, etc.

VMs traditionally get installed at /opt/${P}

We have the concept of a 'system' VM and a 'user' VM. The system VM is
the default VM that will be used for root, and for users who haven't
selected a user VM. The user VM is, as one might guess, selected on a
per user basis. It is worth noting that root always uses the system VM,
and as a result, packages use the system VM when being merged by emerge.

java-config is used to set the system and user VM. When you do so, the
appropriate file from /etc/env.d/java is copied to/etc/env.d/20java for
the system VM or to ~/.gentoo/java-env for the user VM.

java-config's notion of the current VM is tied entirely to the
environment, specifically to JAVA_HOME. Therefore, if you change the
system VM, you'd need to run env-update and then resource /etc/profile.
Likewise, changing the user VM involves sourcing ~/.gentoo/java-env.

The fact that you're tied to the environment is annoying, because as
mentioned, you need re-source the appropriate files. Now imagine you
have a ton of terminals open... you'd have to source the environment
files from each one.

== Packages ==

When a Java package is built, information about it is saved in
/usr/share/${PN}-${SLOT}/package.env (or /usr/share/${PN}/package.env if
SLOT == 0). In particular, the jars that are associated with the package
are recorded, as well as which jars from other packages are used.
java-config can later be used to query for this information.

== Eclasses ==

There are currently 3 eclasses: java, java-pkg, and java-utils.

java.eclass is used for packages which provide a VM.
java-pkg.eclass is used for most Java packages. It provides tools for
querying installed jars, and for installing various Java related files.
java-utils.eclass provides a few utility functions for dealing with Java
stuff.

= The New System =

== The Java Virtual Machine ==
In addition to the concept of a system and a user VM, the new system has
a build VM. As the name implies, the build VM is used for building
packages (instead of the system VM). Sane defaults are defined on a per
platform basis at /usr/share/java-config-2/config/jdk-defaults.conf [3].
The build VM can further be configured by
/etc/java-config-2/build/jdk.conf [4] .

[3]
https://svn.gentooexperimental.org/svn/java/java-config-ng/branches/axxo/config/jdk-defaults-x86.conf
[4]
https://svn.gentooexperimental.org/svn/java/java-config-ng/branches/axxo/config/jdk.conf

For each Java release (ie 1.3, 1.4, 1.5, etc), you can specify which
vendor and version to use at build time.

In addition to being installed to /opt/${P}, VMs also now have a symlink
in /usr/lib/jvm/${PN}-${SLOT}. The purpose of these symlinks is
explained further down.

The user and system VMs are now represented by symlinks pointing to VMs
located in /usr/lib/jvm/. The system VM lives at
/etc/java-config-2/current-system-vm, and the user VM at
~/.gentoo/java-config-2/current-user-vm . Additionally, an environment
variable, GENTOO_VM, can be used to specify the VM used at a given
instance. GENTOO_VM should be the name of a VM located in /usr/lib/jvm.
So with regard to what VM is used, first GENTOO_VM is checked, then the
user VM (for non-root users), and then lastly the system VM.


All the trusty java binaries, ie java, javac, javadoc, jar, etc, now get
wrappers installed into /usr/bin. These are actually symlinks to
/usr/bin/run-java-tool. This is a script which will figure out which
tool was invoked, and then determine which VM to used using the method
mentioned above.

== Packages ==

We now save more information about the build environment at build time
for each package. This information is saved at
/usr/share/${PN}-${SLOT}/package.env. This is facilitate troubleshooting
bugs. Specially, we collect the VM dependency is (ie >=virtual/jdk-1.4),
what -source and -target were used , and which VM. We also keep track of
where javadocs get installed, and where Java sources get installed,
which are useful for integrating into IDEs like netbeans and eclipse.

== Eclasses ==

For each eclass we previously had, we have a updated version of the
eclass, in addition to a few totally new eclasses.

java-utils-2.eclass
	A utility eclass. All utility functions live here (as opposed to being
spread between java-pkg and java-utils). Among other things, there are
functions for figuring out which VM should be used for building, based
on the DEPEND of a package (ie the virtual/jdk atom), and set GENTOO_VM
accordingly.

java-pkg-2.eclass
	The eclass the Java-only packages should inherit. It uses phase hooks
in order to make sure that the environment is setup properly for each phase.

java-pkg-opt-2.eclass
	Similar to java-pkg-2, except this is for packages that have optional
Java support, ie a USE flag. Phase hooks are again used to ensure the
environment is setup properly for each phase.

java-ant-2.eclass
	Packages that build using ant should inherit this eclass. Essentially,
this is eclass will ensure that build.xml files get rewritten such that
javac tasks have the right source and target attributes. In other words,
it makes sure that the right version bytecode gets compiled.

java-vm-2.eclass
	This is the successor to java.eclass, but more aptly name.

= Letting the Systems Coexist =

Originally, the new eclasses were a replacement for the current
eclasses, ie java-pkg-2 used to still be java-pkg, and java-utils-2 used
to be still be java-utils. Additionally, java-config was unslotted, so
the new version of it replaced the current one. Unfortunately, it was
realized that the new system was not a 100% drop-in replacement for the
current one, so we set forth with the task to find a way to migrate from
one to the other.

The method we decided on was to have the concept of having a
'generation' for each system, where the generation consists of the
eclasses, a version of java-config, and the tools that support it. To
cope with java-config, it is now slotted, and installs
/usr/bin/java-config-<insert generation here> . There is then a wrapper
for java-config at /usr/bin/java-config, which reads a variable
WANT_JAVA_CONFIG to determine which java-config to use, or if not
specified, the most recent version. The eclasses then export the
appropriate value of WANT_JAVA_CONFIG to get the right version of
java-config they need.

The existing eclasses and java-config-1.x will be 'generation 1'. The
new eclasses and java-config-2.x will be 'generation 2'.

The first packages to be migrated to generation 2 will be the VMs. When
installed, they will also install environment files that are compatible
with generation 1. This will allow you to have both a system VM for
generation 1, in addition to the system and build VM for generation 2.
The consequence of this is that ebuilds using generation 1 will continue
to build/run as they previously did, while allowing packages to be
migrated to use generation 2 on a package-by-package basis.  So, all
existing ebuilds will continue to work, and will be replaced
incrementally during our migration.

A note on Java 1.5 here... The 1.5 JDKs don't install a generation-1
compatible environment file. This is to prevent the problems already
describe with 1.5, and should prevent them from happening. At this
point, we should be able to unmask the 1.5 JDKs. So 1.5 will be
available, but only to generation-2 packages.

The next packages to be migrated will be ones that need the features of
the new generation. In particular, this means packages which need Java
1.5 to build and/or run.

Once these packages have been migrated, the rest of the tree which uses
Java will be migrated as expediently as possible. I've written some
notes about how to migrate over on our wiki [5]
[5] https://projects.gentooexperimental.org/expj/wiki/Migrating_packages

= Using the new system =

I have written documentation on switching to the new system, from the
user's perspective, over at our wiki [6]

[6]
https://projects.gentooexperimental.org/expj/wiki/Using_migration-overlay

= Feedback =

Obviously, these aren't exactly light changes that need to be made.
Therefore, it's important to get as much feedback as possible, and to
make the upgrade process as smooth as possible.

I'd hope to get these changes put into the tree in the next week or two,
pending feedback. This development has been a long time in the making,
and the Java 1.5 problem has been a thorn in our side for long enough.
-- 
gentoo-dev@g.o mailing list