Re: [gentoo-dev] RFC: BLAS and LAPACK runtime switching (Re-designed)

[Mo Zhou]

Hi Gentoo devs,

I redesigned the solution for BLAS/LAPACK runtime switching.
New solution is based on eselect+ld.so.conf . See following.

> Goal
> ----
>
>   * When a program is linked against libblas.so or liblapack.so
>     provided by any BLAS/LAPACK provider, the eselect-based solution
>     will allow user to switch the underlying library without recompiling
>     anything.

Instead of manipulating symlinks, I wrote a dedicated eselect module
for BLAS:

https://github.com/cdluminate/my-overlay/blob/master/app-eselect/eselect-blas/files/blas.eselect-0.2

This implementation will generate a corresponding ld.so.conf file
on switching and refresh ld.so cache.

advantages:

1. not longer manipulates symlinks under package manager directory,
   see https://bugs.gentoo.org/531842 and https://bugs.gentoo.org/632624

2. we don't have to think about static lib and header switching like
   Debian does.

>   * When a program is linked against a specific implementation, e.g.
>     libmkl_rt.so, the solution doesn't break anything.

This still holds with the new solution.

> Solution
> --------
>
> Similar to Debian's update-alternatives mechanism, Gentoo's eselect
> is good at dealing with drop-in replacements as well. My preliminary

The redesigned solution totally diverted from Debian's solution.

* sci-libs/lapack provides standard API and ABI for BLAS/CBLAS/LAPACK
  and LAPACKE. It provides a default set of libblas.so, libcblas.so,
  and liblapack.so . Reverse dependencies linked against the three
  libraries (reference blas) will take advantage of the runtime
  switching mechanism through USE="virtual-blas virtual-lapack".
  Reverse dependencies linked to specific implementations such as
  libopenblas.so won't be affected at all.

* every non-standard BLAS/LAPACK implementations could be registered
  as alternatives via USE="virtual-blas virtual-lapack". Once the
  virtual-* flags are toggled, the ebuild file will build some
  extra shared objects with correct SONAME.

  For example:

  /usr/lib64/libblis.so.2 (SONAME=libblis.so.2, general purpose)
  /usr/lib64/blas/blis/libblas.so.3 (USE="virtual-blas",
SONAME=libblas.so.3)
  /usr/lib64/blas/blis/libcblas.so.3 (USE="virtual-blas",
SONAME=libcblas.so.3)

* Reverse dependencies of BLAS/LAPACK could optionally provide the
  "virtual-blas virtual-lapack" USE flags.

  if use virtual-*:
      link against reference blas/lapack
  else:
      link against whatever the ebuild maintainer like and get rid
      of the switching mechanism

> Proposed Changes
> ----------------
>
> 1. Deprecate sci-libs/{blas,cblas,lapack,lapacke}-reference from gentoo
>    main repo. They use exactly the same source tarball. It's not quite
>    helpful to package these components in a fine-grained manner. A
> single
>    sci-libs/lapack package is enough.

sci-libs/{blas,cblas,lapack,lapacke}::gentoo should be deprecated. They
are based on exactly the same source tarball, and maintaining 4 ebuild
files for a single tarball is not a good choice IHMO. Those old ebuild
files seems to leverage the flexibility of upstream build system
because it enables one to, for example, skip the reference blas build
and use an existing optimized BLAS impelementation and hence introduce
flexibility. That flexibility is hard to maintain and is not necessary
anymore with the new runtime switching mechanism.

That's why I propose to merge the 4 ebuild into a single one:
sci-libs/lapack. We don't need to add the "reference" postfix
because no upstream will loot the name "lapack". When talking
about "lapack" it's always the reference implementation.

> 2. Merge the "cblas" eselect unit into "blas" unit. It is potentially
>    harmful when "blas" and "cblas" point to different implementations.
>    That means "app-eselect/eselect-cblas" should be deprecated.

eselect-cblas should be deprecated. That affects gsl because it is
registered as an cblas alternative. gsl doesn't provide the standard
BLAS (fortran) API+ABI so it will not be added as a runtime switching
candidate.

Does this redesinged solution look acceptable now?

Best,
Mo.