| 1 |
Hello, all. |
| 2 |
|
| 3 |
Short summary: PMS Test Suite is a suite of tools and a test library |
| 4 |
to test any of the Gentoo Package Managers for compliance against |
| 5 |
the Package Manager Specification. |
| 6 |
|
| 7 |
Homepage: http://www.gentoo.org/proj/en/qa/pms/pms-test-suite.xml |
| 8 |
gitweb: http://git.overlays.gentoo.org/gitweb/?p=proj/pms-test-suite.git |
| 9 |
github (mirror): https://github.com/mgorny/pms-test-suite |
| 10 |
|
| 11 |
A few last changes: |
| 12 |
- small fixes in code, |
| 13 |
- improvements in docs, |
| 14 |
- cleaner (in whitespace) HTML output, |
| 15 |
- --create-repo-only option (for convenience). |
| 16 |
|
| 17 |
The 0.1 release is in gx86 now (as app-portage/pms-test-suite). |
| 18 |
|
| 19 |
The current HTML output: |
| 20 |
- http://www.gentoo.org/proj/en/qa/pms/pms-test-suite-output.html |
| 21 |
|
| 22 |
|
| 23 |
Detailed summary |
| 24 |
---------------- |
| 25 |
|
| 26 |
Right now, PMS Test Suite supplies 21 test cases on PMS compliance |
| 27 |
which expand into 30 tests on the usual run. All three major Package |
| 28 |
Managers (paludis, pkgcore & portage) are supported. Simple CLI result |
| 29 |
output and HTML output is available. |
| 30 |
|
| 31 |
All tests are associated with ranges of EAPIs. The test suite can be |
| 32 |
adjusted to either use random EAPI of each range or all of them |
| 33 |
(comprehensive mode). Test suite can also be set to run tests on EAPIs |
| 34 |
where test results are undefined (to show how PMs actually handle |
| 35 |
those). |
| 36 |
|
| 37 |
All tests are built of smaller 'assertions'. Each assertion has |
| 38 |
associated an equality operator and an (un)expected value. This can |
| 39 |
best be seen on the HTML output where they are used to explain how |
| 40 |
tests proceeded and why they have succeeded or failed. |
| 41 |
|
| 42 |
PMS Test Suite is able to generate all test ebuilds and eclasses |
| 43 |
itself. Using Portage, it can create a temporary repository too while |
| 44 |
other PMs require adding the repository to PM config (as described |
| 45 |
in README). Portage & Paludis backends are able to generate Manifests |
| 46 |
too; Portage & pkgcore backends are able to run without Manifests. |
| 47 |
|
| 48 |
PMS Test Suite is able to run the test suite using multiple Package |
| 49 |
Managers and create a single output on that. This is especially useful |
| 50 |
with HTML output backend which creates PM comparison chart then. |
| 51 |
|
| 52 |
|
| 53 |
Side effects |
| 54 |
------------ |
| 55 |
|
| 56 |
- a number of PMS spec patches improving its clearness, |
| 57 |
- few fixes to pkgcore (0.6.6 failed 4 tests), |
| 58 |
- gentoopm -- a quite extensive, common interface to paludis, pkgcore |
| 59 |
& portage. |
| 60 |
|
| 61 |
As gentoopm is a dependency of PMS Test Suite and it was developed |
| 62 |
in parallel, I will include 0.2.1 version of it in the final GSoC |
| 63 |
zipball. |
| 64 |
|
| 65 |
|
| 66 |
Differences from the initial plan |
| 67 |
--------------------------------- |
| 68 |
|
| 69 |
Well, to be honest the actual timeline was quite different. I was able |
| 70 |
to get the test suite running much earlier than I expected, and thus |
| 71 |
some of the points were done much earlier and a few low-priority ones |
| 72 |
were done a little later. |
| 73 |
|
| 74 |
I expected the Test Suite to take longer to run, and planned wasting |
| 75 |
more time waiting for the results when checking whether it's working |
| 76 |
fine. To be honest, I'd have to thank Brian Harring for pkgcore because |
| 77 |
it helped me testing the test suite a lot faster. |
| 78 |
|
| 79 |
The initial plan assumed the test library would have a pretty specific |
| 80 |
format and a spec describing it. I've actually implemented it using |
| 81 |
Python modules and thus the format is described using epydoc. |
| 82 |
|
| 83 |
|
| 84 |
Future outlook |
| 85 |
-------------- |
| 86 |
|
| 87 |
As I mentioned on my devblog [1], the first important point after GSoC |
| 88 |
is to replace D-Bus IPC by something else. This will also cover |
| 89 |
removing GLib dep and getting Python3 compat. |
| 90 |
|
| 91 |
When no longer depending on D-Bus, the test suite could be reused |
| 92 |
inside our infra to test PMs. Brian has already shown interest in |
| 93 |
reusing it within pkgcore development infra. Another output backend |
| 94 |
will be probably implemented for that. |
| 95 |
|
| 96 |
A long-term plan covers also moving more of PM/ebuild execution code |
| 97 |
from pms-test-suite into gentoopm. |
| 98 |
|
| 99 |
[1]:https://blogs.gentoo.org/mgorny/2011/08/14/pms-test-suite-d-bus-→-…/ |
| 100 |
|
| 101 |
|
| 102 |
Well, I guess that's all. Hopefully see you all next year! |
| 103 |
|
| 104 |
-- |
| 105 |
Best regards, |
| 106 |
Michał Górny |
Archives