[gentoo-soc] Benchmarking suite - Report 11

[Andrea Arteaga]

This is the report of the project "Automated benchmark suite for
numerical libraries in Gentoo" for the week 1 - 7 August.
This is also available on my blog [1].

== Project description ==

The project aims to develop a simple yet powerful automated system of
benchmarking for numerical libraries. The Gentoo software system
provides many implementations of widely used standards such as BLAS,
CBLAS, LAPACK, ScaLAPACK and some other numerical library such as
FFTW, MKL. The developed tools will aid the system maintener to choose
the best suited implementation with respect to the machine hardware
and to test the samer implementation or different ones with different
compilers, compiler versions and compile flags.


== Progrees during the week ==

This week has been devoted to the following features:

* Stabilization of the PBLAS/ScaLAPACK tests
* Introduction of the METIS tests
* Implementation of the LAPACK accuracy tests
* Refinement of the LAPACK performance tests


=== METIS ===
METIS is a package — a set of executables and a library — that perform
some common preliminary numerical work: graph partitioning, mesh
partitioning, sparse matrix reordering. The metis.py module taht has
been introduced benchmarks the run times of the executables "pmetis"
and "kmetis" when doing the most important task: graph partitioning —
the other task rely on this one. In order to keep the benchmarks fair,
the same input data is processed by every “implementation” (i.e. for
each different compilation of the sci-libs/metis package). The module
is already stable and usable, although some more tests could be added.


=== LAPACK performance tests ===
We decided to benchmark matrix decompositions rather than full
solvers, as the decompositions are the real core of each LAPACK
solver, while the substitutions and transposed matrix-vector
multiplication are BLAS tasks that have to be benchmarked separately.
Therefore the following tests have been deprecated — they are still
available, but are not part of the standard set:

* general_solve: General linear system of equations solver
* least_squares: General linear least squares solver
* symm_ev: Symmetric matrix eigensolver — eigenvalues only

Some new tests have been added, and the following are the resulting
standard tests:
* lu_decomp: LU decomposition
* cholesky: Cholesky decomposition of a SPD matrix
* qr_decomp (new): QR decomposition
* svd_decomp (new): Singular Values Decomposition
* syev (new): Symmetric matrix eigensolver (eigenvalues and
eigenvectors), full diagonalization
* stev (new): Tridiagonal matrix eigensolver (eigenvalues and
eigenvectors), full diagonalization


=== LAPACK accuracy tests ===
Following the blas_accuracy.py module strategy, a new much more
interesting module has been added for testing the different LAPACK
implementations for accuracy. This includes every LAPACK standard
tests (see above section). The matrix decomposition are tested by
multiplying the results and comparing this to the original matrix; the
eigensolvers are treated as decompositions (diagonalization) and
tested the same way. Some reports are available on my homepage [2].


== Plan for the next weeks ==

The GSoC programme is reaching the end. Next week is the soft “pencils
down” date. Therefore I will spend the next week doing only tests and
bugfixes, writing documentation and performing other “administrative”
tasks (ebuild refinment, repository management,…). On Friday the
unstable branch of the repository will be merged to the stable. Then a
full report will be written and sent to the mailing list.

After the soft “pencils down” date, only critical bugs will be fixed
and more documentation will be added. Then, I will release a stable,
tested, documented and supported version of my suite just before the
hard “pencils down” date.

I would appreciaty *very* *much* some helping hand in testing my
script! If somebody is interested in having a look of it, it very easy
to install the required packages and run some tests. So please if you
have some time, let it run and report bugs that you find (if any) to
me; if you don’t find bugs, please write to me as well! I will write a
short howto for run the tests and send the link to the mailing list.


Best regards
Andrea Arteaga

[1] http://wp.me/pzWEm-8F
[2] http://www.phys.ethz.ch/~arteagaa/soc/lapack_accuracy/