[gentoo-dev] python-2.3.2 testing required

[Alastair Tse]

Hi All,

I need some more testing for python-2.3.2 before it is released into
~x86. This is a call to Gentoo developers and people interested in
helping to test. 

If you test, make sure you know what you are doing. Portage requires
python. It is best to have a binary package of your current portage and
python, just in case something majorly bad happens.

I would greatly appreciate any feedback on bugs and niggles with the
upgrade process. Report bugs on bugs.gentoo.org please. I'll list some
of the changes, points of interest and errata.

How to Upgrade:
===============
1. Unmask _both_ =dev-lang/python-2.3* and >=sys-apps/portage-2.0.49-r16
in package.mask.
2. run: emerge -u portage python 
(note that -r16 should work with both 2.2 and 2.3)
3. run: /usr/portage/dev-lang/python/python-updater

Major Changes:
==============
Unicode support has a choice of UCS2 or UCS4. 
---------------------------------------------

UCS4 uses significantly more memory than UCS2. For those who are not
familiar with unicode, UCS2 means representing unicode characters with
16 bit words and UCS4 means representing unicode characters in 32 bit
words. Not that both Redhat (>9) and Debian (unstable) have python-2.3
compiled with UCS4 by default.

Currently, you get UCS4 if you have "cjk" in your USE flags because the
only language to use the extra pane in UCS4 are CJK langauges. I've been
using UCS4 for a while now and it works without any problems. 

The reason why I'm not making this default is because UCS4 python uses
more memory. An example is supybot (Python IRC bot) that uses 8M for
UCS2 and 13M for UCS4. But note that this example is not scientific
because the machines were different in kernel version, compiler and
compiler optimisations.

I'm willing to listen to any ideas about whether to support UCS4 by
default or not.

The only problem that may occur is if you change between UCS2 and UCS4,
you will need to recompile any external python codec packages like
japanesecodec, koreancodec, cjkcodecs, iconvcodec.

For those who want to know more about UCS and Unicode, you maybe
interested in some articles like:
http://uche.ogbuji.net/tech/akara/nodes/2003-01-01/characters
http://www.python.org/cgi-bin/faqw.py?req=show&file=faq04.107.htp
http://www.debian.org/doc/manuals/intro-i18n/ch-codes.en.html#s-unicodes-ccs

Handling *.pyc *.pyo
---------------------

The only other major change in the handling of python is with
byte-compiled modules. With previous versions of python, portage had a
problem tracking *.pyc and *.pyo files. The reason is that these files
were registered with portage at install time, but may be modified at
runtime if either the timestamp on the python executable has changed or
the version has been upgraded.

This traditionally created two problems. Firstly, *.pyc and *.pyo were
left behind because their timestamps and md5sum didn't match when they
were installed[1]. The second problem is that sometimes python gets run
inside sandbox and suddenly requests certain .pyc and .pyo to be
re-compiled, creating sandbox violations. You will notice special
workarounds in libsandbox to avoid this problem.

The solution I am now employing is the not let portage track the .pyc
and .pyo, and generating them at pkg_postinst(). For package removals, a
pkg_postrm(), orphaned *.pyc *.pyo (orphaned means the .pyc that don't
have a corresponding .py) will be removed. The other problem with python
recompiling modules at runtime is now solved by setting an environment
variable (PY_DONTCOMPILE) during merging that prevents python from
attempting to recompile modules.

This is now implemented in python.eclass and distutils.eclass. If you
maintain anything that creates python modules, you should either read
the eclass to find out what needs to change for python-2.3 or email me
the packages that need to be looked at. Most packages that use distutils
should work automatically with this new scheme.

[1] .. http://bugs.gentoo.org/show_bug.cgi?id=8804

Python Updater
--------------
A neccessary evil for the upgrade is to recompile all the modules that
depend on Python. I use a very simple rule to determine that. The rule
is if a package has a file in /usr/lib/python2.2, then it will need to
be re-emerged.

There is some special magic that does that in
/usr/portage/dev-lang/python/files/python-updater which is able to get
all the packages, sort the dependencies to avoid merging problems and
output a log file to /tmp/python-updater.log.

I'd appreciate as much feedback on this as possible. Especially on
whether I should run it in pkg_postinst() to make sure people who don't
read the trailing einfo's actually upgrade their python modules. It
might be technically unsound for me to run it in pkg_postinst() (nested
emerge's sound like a bad idea.)

Other Python Improvements
--------------------------
* New version, supposedly faster by 30%.
* bsddb support for 4.x
* Supports generators (very spiffy for implementing weightless threads)
* A new idle (python gui console)

Known Problems:
===============
* I haven't tried this on a install from scratch. Maybe someone can help
me to create a stage with only 2.3 and see how things work.

* Even though python-2.3.2 requires portage-2.0.49-r16, I can't put a
dep in otherwise it may create a circular dependency. Probably the best
solution would be to have a pkg_setup check before it goes ahead.

* On some machines, portageq doesn't work after python upgrade. Make
sure you are using the latest python version.
(http://bugs.gentoo.org/show_bug.cgi?id=32374)

Plan for Python's Future
========================
* I'm aiming to push Python-2.3.2 to ~x86 in the next week or so. There
are most likely packages that don't work under 2.3 that need to be
upgraded. If you find one, please report it.

* Maybe we should think about using a non version specific directory for
Python modules to ease upgrade.

That's all folks!

Thanks,
-- 
Alastair 'liquidx' Tse
 >> Gentoo Developer
 >> http://www.liquidx.net/ | http://dev.gentoo.org/~liquidx/