[gentoo-dev] Support for multiple ABIs (for Python modules etc.)

[Arfrever Frehtes Taifersar Arahesis]

I would like to present the plan of support for multiple ABIs. It should be sufficient for
Python modules and might be also appropriate for some other ABI types (e.g. for Ruby modules).

1. Portage will support repository-specific, profile-independent ${REPOSITORY}/profiles/settings
configuration files (filename can be changed). They will set some variables (similarly to
make.defaults files). These variables shouldn't be overrideable by profiles, eclasses, ebuilds
or by user's configuration files (e.g. /etc/portage/*). For future compatibility these files
could also support lists/indexed arrays and dictionaries/associative arrays/maps.

2. ${REPOSITORY}/profiles/settings will set ABI_TYPES and ${ABI_TYPE^^}_ABI_SUPPORTED_VALUES variables.
ABI_TYPES defines supported ABI types. ${ABI_TYPE^^}_ABI_SUPPORTED_VALUES defines supported ABIs
for given ABI type.

  Example (${REPOSITORY}/profiles/settings):
    ABI_TYPES="python ruby ..."
    PYTHON_ABI_SUPPORTED_VALUES="2.4 2.5 2.6 3.0 3.1"
    RUBY_ABI_SUPPORTED_VALUES="1.8 1.9"

3. make.defaults files will support ${ABI_TYPE^^}_ABI_FORCED_VALUES/${ABI_TYPE^^}_ABI_MASKED_VALUES
variables to force/mask specific ABIs in given profiles.

  Example (make.defaults):
    PYTHON_ABI_MASKED_VALUES="2.4 2.5"
    PYTHON_ABI_FORCED_VALUES="3.1"

4. For convenience of developers ABI dependencies will be able to be specified in explicit and
slightly implicit way. Ebuilds/eclasses, which support multiple ABIs, will set USED_ABI_TYPES.
Ebuilds/eclasses will be able to set RESTRICT_${ABI_TYPE^^}_ABIS variables which will support
'*' wildcard. USED_ABI_TYPES and RESTRICT_${ABI_TYPE^^}_ABIS variables should be cumulative
(across eclasses and ebuild).

  Example (ebuild):
    USED_ABI_TYPES="python"
    # This package doesn't work yet with Python 2.6 and 3.*
    RESTRICT_PYTHON_ABIS="2.6 3*"

4.1. Implicitly specified ABI dependencies. During calculation of dependencies of given package,
Portage will verify if all dependencies, which use given ABI type, have been built with enabled
support for these ABIs, which are enabled for given package.

  Example (ebuild):
    # dev-python/bsddb3
    USED_ABI_TYPES="python"
    RDEPEND=">=sys-libs/db-4.6"
    DEPEND="${RDEPEND}
        dev-python/setuptools
        doc? ( dev-python/sphinx )"

    Assuming that dev-python/setuptools and dev-python/sphinx set USED_ABI_TYPES="python",
    Portage will verify that, when user runs `USE_PYTHON="2.5 2.6" emerge dev-python/bsddb3`,
    dev-python/setuptools and dev-python/sphinx (when USE="doc" is enabled) are built with
    enabled support for at least "2.5" and "2.6" Python ABIs. If they are already installed
    without support for e.g. "2.5" Python ABI, Portage will print appropriate error message
    (similarly to unsatisfied USE dependencies) and exit.

4.2. Explicitly specified ABI dependencies. {,P,R}DEPEND variables will support specifying
ABI dependencies in explicit way. {,P,R}DEPEND variables will also support ABI conditionals.
I suggest using {ABI_type[comma-delimited values]} syntax, but it can be changed.

  Example (ebuild):
    DEPEND="category/package1{python[2.5,2.6],ruby[-1.8]}"
    RDEPEND="{python[2.6]}? ( category/package2 )

  Example (python.eclass):
    DEPEND="
        {python[2.4]}? ( dev-lang/python:2.4 )
        {python[2.5]}? ( dev-lang/python:2.5 )
        {python[2.6]}? ( dev-lang/python:2.6 )
        {python[3.0]}? ( dev-lang/python:3.0 )
        {python[3.1]}? ( dev-lang/python:3.1 )
    "
    RDEPEND="${RDEPEND}"

5. 18th line of metadata cache files will contain the list of ABI types and values supported
(i.e. not restricted by RESTRICT_${ABI_TYPE^^}_ABIS variables) by given package.

  Example (18th line of metadata cache file):
    python[2.5,2.6,3.0,3.1] ruby[1.8,1.9]

Every change to ${REPOSITORY}/profiles/settings should probably invalidate whole metadata
cache of given repository.

6. Portage will support USE_${ABI_TYPE^^} variables in /etc/make.conf to specify which ABIs
should be enabled.

  Example (/etc/make.conf):
    USE_PYTHON="2.5 2.6 3.1"
    USE_RUBY="1.8 1.9"

7. Portage will set ${ABI_TYPE^^}_ABIS variables in ebuild environment. ${ABI_TYPE^^}_ABIS
should contain ABIs forced by ${ABI_TYPE^^}_ABI_FORCED_VALUES and these ABIs enabled by user
in /etc/make.conf by USE_${ABI_TYPE^^} which aren't masked by ${ABI_TYPE^^}_ABI_MASKED_VALUES.

  Example (ebuild):
    src_compile() {
        for abi in ${PYTHON_ABIS}; do
            emake PYTHON=$(get_PYTHON ${abi})
        done
    }

Ebuilds/eclasses should implicitly enable support for one ("default") ABI, when user hasn't
set appropriate USE_${ABI_TYPE^^} variable. For example Python modules would use ABI of
/usr/bin/python.

8. Portage will store the list of supported ABIs in /var/db/pkg/${CATEGORY}/${PF}/SUPPORTED_ABIS
and the list of enabled ABIs in /var/db/pkg/${CATEGORY}/${PF}/ABIS.

  Example (/var/db/pkg/${CATEGORY}/${PF}/SUPPORTED_ABIS):
    python[2.4,2.5,2.6,3.0,3.1] ruby[1.8,1.9]

  Example (/var/db/pkg/${CATEGORY}/${PF}/ABIS):
    python[2.6,3.1] ruby[1.9]

Names/syntax of specific variables etc. can be changed. Please write constructive comments.

-- 
Arfrever Frehtes Taifersar Arahesis