[gentoo-soc] Benchmarking suite - Report 7

[Andrea Arteaga]

Hello all.
This week the automated benchmarking suite project has received much
work. During the first half of the week I

* improved the HTML report
* closed a number of bugs
* implemented a module for testing FFTW

Then, I spent a couple of days investigating the distributed-memory
parallel routines of BLACS, PBLAS and ScaLAPACK, and their
availability in Gentoo. I also set up an MPI environment under Gentoo
and decided to share my experience within the documentation of Gentoo,
which is slightly outdated.

Now, some work is being done in the pkg-config interface of the
script. Pkg-config is used by the script to retrieve the compile
instructions when compiling and linking against the numerical
libraries. The new alternatives-2 system relies on this useful tool,
and the benchmarking project also does. Some refinement will allow the
user to customize even more the tests.


Implementing a module for testing FFTW
========================================

This field is a bit special, as FFTW is not a standard which is
reimplemented in optimized libraries (which is the case of BLAS,
CBLAS, LAPACK,...). However, it is useful to compare the speed of FFTW
compiled with different compilers, versions or flags, so there is a
particular module for testing this widely used library.

FFTW can perform a wide number of tasks, and a survey has been taken
[1] asking the community about the most wanted tests. Following the
results, the following tests have been implemented:
* Forward Discrete Fourier Transormation
* Backward Discrete Fourier Transormation
* The same two with the flag FFTW_MEASURE instead of FFTW_ESTIMATE

After the mid-term evaluation also 2-dimensional tests will be
provided. Now this four tests are available and implemented within the
BTL framework, which makes the benchmarking suite robust. The tests
until now have been successful.


Next week's milestones
========================

The mid-term evaluation will come soon. For this, I'm performing many
tests in order to present, as mid-term result, a general comprehensive
report which makes clear that the project is already usable. My mentor
reported that my script already allowed him to find some bugs in a
package, which already is a good results for me -- even if the purpose
of the script is not (only) finding bugs --. My mid-term report will
hopefully persuade many people to run some benchmarks on their
computers.

So, objectives for the next week are:

* Improve the pkg-config interface
* Improve the user configuration possibilities
* Improve the niceness and information of the HTML reports
* Test the suite as much as possible in order to generate a
comprehensive set of reports for each supported library (BLAS, CBLAS,
LAPACK, FFTW)


Best regards
Andrea Arteaga


[1] https://spreadsheets.google.com/spreadsheet/viewform?hl=en_US&formkey=dGlKbjBfLW5uaGt4QnQxNFJYUVp1QlE6MQ#gid=0