Re: XML and unit testing

[Lars Marius Garshol <larsga@garshol.priv.no>]

* David Brownell
| 
| Another testing framework to think about is Mauve.

I've had a brief look at Mauve before, but came away thinking it was a
collection of test cases for the Java core libraries and nothing more.
Apparently I was too quick. What else does Mauve contain but a
collection of test cases?

| I'm most familiar with separate tests, plugging into some sort of
| testing framework or harness.  The harness automates running lots of
| tests, setting each one up and collecting results
| (pass/fail/expected/...) as well as (important!) organizing
| unexpected results for analysis.

I've been using JUnit to create a test suite consisting of by now
about 650 test cases and am not really very satisfied with it. One
unmet wish is that it should be easy to collect test cases into
hierarchical groups, and also to decide which groups to run and which
to skip. I've had to build utilities on top of JUnit to be able to
achieve this.

Another is that quite often one may want to have test cases generated
dynamically based on things outside the test code itself, such as
input samples. This usually means doing regression testing with JUnit,
but I feel it makes sense to have both unit tests and regression tests
in the same framework.

JUnit really does not support this at all well, and although we've
made it work I can't say I'm thrilled with the results.

A third problem is that with JUnit it seems tricky to reuse the same
set of test cases for different implementations of an interface. We've
worked around it by using our mechanism for dynamically generating
test cases, but again the result was less than great.

With PyUnit the first and third problems were much easier to deal
with, since one could make use of Python's extremely convenient
reflection and multiple inheritance.

The fourth problem is that the UIs seem to be designed in such a way
that programmatically controlling how test cases are gathered and
which ones are run does not seem to be possible.

Does anyone know of solutions to these problems within JUnit, or
better testing frameworks that allow good solutions?

| API specs need to be written to be testable.  

Very true. APIs designed the right way are easy to write test cases
for, but many APIs make automated testing near-impossible. Not having
well-defined boundaries and having too many internal dependencies
often causes this (in addition to the problem causes you mentioned).

| There's also "regression testing", which you can think of as unit
| testing that's maintained in sync with the software so that you can
| know updates aren't going to break things.

I've used regression testing extensively with SAX applications and
been very happy with the results. Having a highly configurable API for
the applications makes this more difficult, of course, but it should
still be a practicable approach, I think.

--Lars M.