Re: [gentoo-dev] Changes to the way Java packages are built

[Joshua Nichols]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Joshua Nichols wrote:
> = Background =
>
> As some might have noticed, Java 1.5 has been package.masked for some
> time now. There are a number of issues introduced with 1.5 that have
> kept it in package.mask. Please see the Java 1.5 FAQ [1] for more details.
>
> [1] http://www.gentoo.org/proj/en/java/tiger-faq.xml
>
> About a year ago, work was begun on improving our part of the build
> system (read: Java related eclasses and our java-config tool) in a way
> to make it much more flexible in general, but specifically improve it to
> get around the known issues. It took about six months to fully develop.
> Unfortunately, the new system was not quite a drop-in replacement. So,
> it took from then until now to determine how to migrate from the current
> system  to the new one in a sane way.
>
>
> = The Current System =
>
> To give some proper background, it is worth going over the current
> system briefly.
>
> == The Java Virtual Machine ==
> Each Java Virtual Machine (VM) installs an environment file into
> /etc/env.d/java. These files contain information about where JAVA_HOME
> is, the PATH to include, etc.
>
> VMs traditionally get installed at /opt/${P}
>
> We have the concept of a 'system' VM and a 'user' VM. The system VM is
> the default VM that will be used for root, and for users who haven't
> selected a user VM. The user VM is, as one might guess, selected on a
> per user basis. It is worth noting that root always uses the system VM,
> and as a result, packages use the system VM when being merged by emerge.
>
> java-config is used to set the system and user VM. When you do so, the
> appropriate file from /etc/env.d/java is copied to/etc/env.d/20java for
> the system VM or to ~/.gentoo/java-env for the user VM.
>
> java-config's notion of the current VM is tied entirely to the
> environment, specifically to JAVA_HOME. Therefore, if you change the
> system VM, you'd need to run env-update and then resource /etc/profile.
> Likewise, changing the user VM involves sourcing ~/.gentoo/java-env.
>
> The fact that you're tied to the environment is annoying, because as
> mentioned, you need re-source the appropriate files. Now imagine you
> have a ton of terminals open... you'd have to source the environment
> files from each one.
>
> == Packages ==
>
> When a Java package is built, information about it is saved in
> /usr/share/${PN}-${SLOT}/package.env (or /usr/share/${PN}/package.env if
> SLOT == 0). In particular, the jars that are associated with the package
> are recorded, as well as which jars from other packages are used.
> java-config can later be used to query for this information.
>
> == Eclasses ==
>
> There are currently 3 eclasses: java, java-pkg, and java-utils.
>
> java.eclass is used for packages which provide a VM.
> java-pkg.eclass is used for most Java packages. It provides tools for
> querying installed jars, and for installing various Java related files.
> java-utils.eclass provides a few utility functions for dealing with Java
> stuff.
>
> = The New System =
>
> == The Java Virtual Machine ==
> In addition to the concept of a system and a user VM, the new system has
> a build VM. As the name implies, the build VM is used for building
> packages (instead of the system VM). Sane defaults are defined on a per
> platform basis at /usr/share/java-config-2/config/jdk-defaults.conf [3].
> The build VM can further be configured by
> /etc/java-config-2/build/jdk.conf [4] .
>
> [3]
> https://svn.gentooexperimental.org/svn/java/java-config-ng/branches/axxo/config/jdk-defaults-x86.conf
> [4]
> https://svn.gentooexperimental.org/svn/java/java-config-ng/branches/axxo/config/jdk.conf
>
> For each Java release (ie 1.3, 1.4, 1.5, etc), you can specify which
> vendor and version to use at build time.
>
> In addition to being installed to /opt/${P}, VMs also now have a symlink
> in /usr/lib/jvm/${PN}-${SLOT}. The purpose of these symlinks is
> explained further down.
>
> The user and system VMs are now represented by symlinks pointing to VMs
> located in /usr/lib/jvm/. The system VM lives at
> /etc/java-config-2/current-system-vm, and the user VM at
> ~/.gentoo/java-config-2/current-user-vm . Additionally, an environment
> variable, GENTOO_VM, can be used to specify the VM used at a given
> instance. GENTOO_VM should be the name of a VM located in /usr/lib/jvm.
> So with regard to what VM is used, first GENTOO_VM is checked, then the
> user VM (for non-root users), and then lastly the system VM.
>
>
> All the trusty java binaries, ie java, javac, javadoc, jar, etc, now get
> wrappers installed into /usr/bin. These are actually symlinks to
> /usr/bin/run-java-tool. This is a script which will figure out which
> tool was invoked, and then determine which VM to used using the method
> mentioned above.
>
> == Packages ==
>
> We now save more information about the build environment at build time
> for each package. This information is saved at
> /usr/share/${PN}-${SLOT}/package.env. This is facilitate troubleshooting
> bugs. Specially, we collect the VM dependency is (ie >=virtual/jdk-1.4),
> what -source and -target were used , and which VM. We also keep track of
> where javadocs get installed, and where Java sources get installed,
> which are useful for integrating into IDEs like netbeans and eclipse.
>
> == Eclasses ==
>
> For each eclass we previously had, we have a updated version of the
> eclass, in addition to a few totally new eclasses.
>
> java-utils-2.eclass
> 	A utility eclass. All utility functions live here (as opposed to being
> spread between java-pkg and java-utils). Among other things, there are
> functions for figuring out which VM should be used for building, based
> on the DEPEND of a package (ie the virtual/jdk atom), and set GENTOO_VM
> accordingly.
>
> java-pkg-2.eclass
> 	The eclass the Java-only packages should inherit. It uses phase hooks
> in order to make sure that the environment is setup properly for each phase.
>
> java-pkg-opt-2.eclass
> 	Similar to java-pkg-2, except this is for packages that have optional
> Java support, ie a USE flag. Phase hooks are again used to ensure the
> environment is setup properly for each phase.
>
> java-ant-2.eclass
> 	Packages that build using ant should inherit this eclass. Essentially,
> this is eclass will ensure that build.xml files get rewritten such that
> javac tasks have the right source and target attributes. In other words,
> it makes sure that the right version bytecode gets compiled.
>
> java-vm-2.eclass
> 	This is the successor to java.eclass, but more aptly name.
>
> = Letting the Systems Coexist =
>
> Originally, the new eclasses were a replacement for the current
> eclasses, ie java-pkg-2 used to still be java-pkg, and java-utils-2 used
> to be still be java-utils. Additionally, java-config was unslotted, so
> the new version of it replaced the current one. Unfortunately, it was
> realized that the new system was not a 100% drop-in replacement for the
> current one, so we set forth with the task to find a way to migrate from
> one to the other.
>
> The method we decided on was to have the concept of having a
> 'generation' for each system, where the generation consists of the
> eclasses, a version of java-config, and the tools that support it. To
> cope with java-config, it is now slotted, and installs
> /usr/bin/java-config-<insert generation here> . There is then a wrapper
> for java-config at /usr/bin/java-config, which reads a variable
> WANT_JAVA_CONFIG to determine which java-config to use, or if not
> specified, the most recent version. The eclasses then export the
> appropriate value of WANT_JAVA_CONFIG to get the right version of
> java-config they need.
>
> The existing eclasses and java-config-1.x will be 'generation 1'. The
> new eclasses and java-config-2.x will be 'generation 2'.
>
> The first packages to be migrated to generation 2 will be the VMs. When
> installed, they will also install environment files that are compatible
> with generation 1. This will allow you to have both a system VM for
> generation 1, in addition to the system and build VM for generation 2.
> The consequence of this is that ebuilds using generation 1 will continue
> to build/run as they previously did, while allowing packages to be
> migrated to use generation 2 on a package-by-package basis.  So, all
> existing ebuilds will continue to work, and will be replaced
> incrementally during our migration.
>
> A note on Java 1.5 here... The 1.5 JDKs don't install a generation-1
> compatible environment file. This is to prevent the problems already
> describe with 1.5, and should prevent them from happening. At this
> point, we should be able to unmask the 1.5 JDKs. So 1.5 will be
> available, but only to generation-2 packages.
>
> The next packages to be migrated will be ones that need the features of
> the new generation. In particular, this means packages which need Java
> 1.5 to build and/or run.
>
> Once these packages have been migrated, the rest of the tree which uses
> Java will be migrated as expediently as possible. I've written some
> notes about how to migrate over on our wiki [5]
> [5] https://projects.gentooexperimental.org/expj/wiki/Migrating_packages
>
> = Using the new system =
>
> I have written documentation on switching to the new system, from the
> user's perspective, over at our wiki [6]
>
> [6]
> https://projects.gentooexperimental.org/expj/wiki/Using_migration-overlay
>
> = Feedback =
>
> Obviously, these aren't exactly light changes that need to be made.
> Therefore, it's important to get as much feedback as possible, and to
> make the upgrade process as smooth as possible.
>
> I'd hope to get these changes put into the tree in the next week or two,
> pending feedback. This development has been a long time in the making,
> and the Java 1.5 problem has been a thorn in our side for long enough.

My apologies if I didn't make it clear, but I should point out that
these changes are slightly disruptive to the end-user. By this, I mean
there are actions that the user would need to take aside from just doing
an emerge world. The steps needed are outlined in the
using_migration-overlay wiki page [1]

[1]
https://projects.gentooexperimental.org/expj/wiki/Using_migration-overlay

So... I really would like as much feedback in particular on switching
over to the new system from the user's perspective. I'd like to get the
process as streamlined as to suppress as much pain and suffering as
possible...

Regards,

Josh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFElzzD8ATTzZyw6sMRAnbrAJ9dmcuXTNeKeD8DjhwSNlEDAGUHRQCfbIrQ
DfNSwV4XbWbU+69o2dxshMY=
=YH4V
-----END PGP SIGNATURE-----
-- 
gentoo-dev@g.o mailing list