Re: [gentoo-dev] RFC: BLAS and LAPACK runtime switching (Re-designed)

[Michał Górny]

On Thu, 2019-06-13 at 00:15 -0700, Mo Zhou wrote:
> Hi Gentoo devs,
>
> I redesigned the solution for BLAS/LAPACK runtime switching.
> New solution is based on eselect+ld.so.conf . See following.
>
> > Goal
> > ----
> >
> >   * When a program is linked against libblas.so or liblapack.so
> >     provided by any BLAS/LAPACK provider, the eselect-based solution
> >     will allow user to switch the underlying library without recompiling
> >     anything.
>
> Instead of manipulating symlinks, I wrote a dedicated eselect module
> for BLAS:
>
> https://github.com/cdluminate/my-overlay/blob/master/app-eselect/eselect-blas/files/blas.eselect-0.2
>
> This implementation will generate a corresponding ld.so.conf file
> on switching and refresh ld.so cache.
>
> advantages:
>
> 1. not longer manipulates symlinks under package manager directory,
>    see https://bugs.gentoo.org/531842 and https://bugs.gentoo.org/632624
>
> 2. we don't have to think about static lib and header switching like
>    Debian does.
>
> >   * When a program is linked against a specific implementation, e.g.
> >     libmkl_rt.so, the solution doesn't break anything.
>
> This still holds with the new solution.
>
> > Solution
> > --------
> >
> > Similar to Debian's update-alternatives mechanism, Gentoo's eselect
> > is good at dealing with drop-in replacements as well. My preliminary
>
> The redesigned solution totally diverted from Debian's solution.
>
> * sci-libs/lapack provides standard API and ABI for BLAS/CBLAS/LAPACK
>   and LAPACKE. It provides a default set of libblas.so, libcblas.so,
>   and liblapack.so . Reverse dependencies linked against the three
>   libraries (reference blas) will take advantage of the runtime
>   switching mechanism through USE="virtual-blas virtual-lapack".
>   Reverse dependencies linked to specific implementations such as
>   libopenblas.so won't be affected at all.
>
> * every non-standard BLAS/LAPACK implementations could be registered
>   as alternatives via USE="virtual-blas virtual-lapack". Once the
>   virtual-* flags are toggled, the ebuild file will build some
>   extra shared objects with correct SONAME.
>
>   For example:
>
>   /usr/lib64/libblis.so.2 (SONAME=libblis.so.2, general purpose)
>   /usr/lib64/blas/blis/libblas.so.3 (USE="virtual-blas",
> SONAME=libblas.so.3)
>   /usr/lib64/blas/blis/libcblas.so.3 (USE="virtual-blas",
> SONAME=libcblas.so.3)
>
> * Reverse dependencies of BLAS/LAPACK could optionally provide the
>   "virtual-blas virtual-lapack" USE flags.
>
>   if use virtual-*:
>       link against reference blas/lapack
>   else:
>       link against whatever the ebuild maintainer like and get rid
>       of the switching mechanism
>
> > Proposed Changes
> > ----------------
> >
> > 1. Deprecate sci-libs/{blas,cblas,lapack,lapacke}-reference from gentoo
> >    main repo. They use exactly the same source tarball. It's not quite
> >    helpful to package these components in a fine-grained manner. A
> > single
> >    sci-libs/lapack package is enough.
>
> sci-libs/{blas,cblas,lapack,lapacke}::gentoo should be deprecated. They
> are based on exactly the same source tarball, and maintaining 4 ebuild
> files for a single tarball is not a good choice IHMO. Those old ebuild
> files seems to leverage the flexibility of upstream build system
> because it enables one to, for example, skip the reference blas build
> and use an existing optimized BLAS impelementation and hence introduce
> flexibility. That flexibility is hard to maintain and is not necessary
> anymore with the new runtime switching mechanism.
>
> That's why I propose to merge the 4 ebuild into a single one:
> sci-libs/lapack. We don't need to add the "reference" postfix
> because no upstream will loot the name "lapack". When talking
> about "lapack" it's always the reference implementation.

What's the real gain here, and how does it compare to loss of
flexibility of being able to build only what the package in question
needs?


-- 
Best regards,
Michał Górny