[gentoo-soc] Benchmarking suite - Final Report

[Andrea Arteaga]

This is the final report of the project “Automated benchmark suite for
numerical libraries in Gentoo” for the Google Summer of Code 2011.
This report is also available on my blog at the address
http://wp.me/pzWEm-8Z.


== Project description ==
The project aims to develop a simple yet powerful automated system of
benchmarking for numerical libraries. The Gentoo software system
provides many implementations of widely used standards such as BLAS,
CBLAS, LAPACK, ScaLAPACK and some other numerical library such as
FFTW, MKL. The developed tools will aid the system maintener to choose
the best suited implementation with respect to the machine hardware
and to test the same implementation or different ones with different
compilers, compiler versions and compile flags.


== Release ==
This report refers to the 0.1 release, which is the commit tagged with
“0.1″ on the git repository. [1]
An ebuild for the 0.1 version will come soon.


== Archieved objectives ==
In the original project description [2], I pointed out that I would
have delivered a benchmarking suite and a script, acting respectively
as “data provider” for the benchmarking results and as system
organizer for the work to do, user interface, base for the
customization, … During the summer I decided to adopt the BTL (Bench
Template Library) as benchmarking suite, and actually much work has
been devoted to this; the BTL was initially designed to be extensible,
and I actually used almost all extensible features of the BTL in order
to adapt it for my purposes. In facts, I discovered that the BTL is
not quite perfect and decided that, after the GSoC, I will spend some
time trying to figure out how this very powerful library could improve
even more. Anyway, it has performed well for my GSoC project, and the
modified version is available on my hg repository. [3]

The script is in facts a Python library that:
* Uses the featured provided by portage to search, compile, emerge
packages in a separate root; resolves the dependencies (this part is
still unstable); follows the user’s instruction on the environment to
use during the emerge process; stores the generated binary packages
for future usage (instruction to the user are printed at the end).
* Writes everything onto comprehensive and organized logs.
* Interprets an highly customizable configuration file provided by the
user to define the tests that have to be run.
* Compiles the benchmarking suite with the correct options, flags, libraries,…
* Wraps the suite execution and provides user-friendly output; stores
its results in an organized fashion.
* Collects the results and plots them.
* Saves the plots and generates a comprehensive HTML report; the
report contains, along with the images, information about the system
and the time, the logs, the configuration file and optionally a
summary figure.
* Cleans the system.

The script is modular: for each different library a module has to be
provided. The module specifies where the particular benchmarking suite
main source is, how to compile it, how to run it, how to interpret the
results,… As most of the modules are tested through the modified BTL
library, a generic code is present in an abstract BTL module and the
specific modules can just inherit this module and add a few
information. The following modules make use of the BTL:
* blas
* cblas
* lapack
* scalapack
* fftw

The following modules make instead use of a different benchmarking
suite, that is now part of the modified BTL, but follows a different
implementation paradigm:
* blas_accuracy
* lapack_accuracy

These test the accuracy of the implementations instead of the
computational speed.

The following module does not use a benchmarking suite, but relies on
the information provided by the executable that is contained in the
package:
* metis

A total of 8 modules are provided. This exceeds the initial
expectations; to be honest, I have to say that some of them (in
particular blas_accuracy and metis) are very basic modules and lack of
some features, while the module scalapack was not tested much and is
to be considered unstable.


== Documentation ==
A web page is present at http://soc.dev.gentoo.org/~spiros/. Here some
documentation about how to install the numbenc package is present.
Since numbench is much Gentoo-specific, only instruction on how to
install it on Gentoo are provided. In facts, it is completely useless
without the emerge and equery commands. The page also gives an
overview on hot to run the script.

The package installs a man page numbench(1) that explains in more
detail how to configure and run a test, where the logs and the results
are and gives some more information. A set of sampl configuration
files come with the package, too, and are installed into
/usr/share/numbench/samples.

The source (which is available on the auto-numerical-bench git
repository [4] for the script and on the cited mercurial repository
for the benchmarking suite [3]) contains some comments that could be
useful for developers that want to write new modules or improve the
project. In any case, if you plan adding features, please contact me!


== License ==
The modified BTL has not changed license and is therefore re-released
with the GPL -2 license. The same license is adopted for the Python
part.


== Results ==
On the web page a set of results are available.[5] They cover almost
all available module and also give examples of configuration files.


== Acknowledgements ==
First of all, thanks to Google for sponsorizing the such a programme.
This is of course useful for the students that have the possibility to
do a real-work job (as student I often feel the need of a similar
experience in my academic activities), earn something and, above all,
at least for us Gentoo students, to get in touch with the FLOSS
community.

Thanks to Donnie and the metors that make Gentoo being a perfect
choice for a student who wants to participate to the SoC.

Many thanks to the whole Gentoo community that is always responsive
and helpful. The forums, IRC channels and mailing-lists are full of
experts that are ready to help you just as you need. Being a SoC
student make you feel like a privileged and very respected person,
which is just wonderful.

And a very big thank to my mentor Sébastien, who helped me a lot
almost daily during the whole summer. Working with Sébastien is very
motivating and his testing work has been very helpful. Thanks very
much!


That’s all!
Best regards
Andrea Arteaga






[1] http://git.overlays.gentoo.org/gitweb/?p=proj/auto-numerical-bench.git;a=tree;h=40c901a4c2be8bf89e20a45e29130f8488176923;hb=587860cfcdc6845385bc89a7e49bca90caaff4a1
[2] http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/spiros/1
[3] https://bitbucket.org/spiros/btl
[4] http://git.overlays.gentoo.org/gitweb/?p=proj/auto-numerical-bench.git;a=summary
[5] http://soc.dev.gentoo.org/~spiros/Results/