XPlorations
| Semantics of Fit: A Path Toward New Tools |
June, 2005
|
|
Fit's standard interpretation tells us how well a program does against a
set of test cases. We can design new semantics for reporters (that give
us interesting information) and for rewriters (that make
interesting transformations
of our tests). |
Semantics of Fit
The Standard Interpretation
A Fit test looks like a table (or a series of tables) in an HTML
document:
| Calculator |
| x |
y |
plus() |
times() |
| 2 |
1 |
3 |
2 |
| 0 |
3 |
3 |
0 |
| 2 |
-1 |
1 |
-2 |
| fit.ActionFixture |
| start |
ScientificCalculator |
|
| enter |
value |
2 |
| press |
plus |
|
| enter |
value |
1 |
| press |
equals |
|
| check |
value |
3 |
But what does a Fit test mean?
This is almost a trick question. One answer is: whatever people agree it
means. Another answer is: whatever the Fit framework, the fixtures, and the
application make it mean. These two things don't always line up,
unfortunately.
Even when we think we're talking about the same interpretation, there can
be differences in what a test means. One version of the Calculator
fixture could
work with the business objects; another could work at a presentation layer;
another could work by manipulating the user interface. These fixtures
might find
different types of defects; they might be easier or harder to write;
and so on.
These all generally assume a standard interpretation: use Fit to
run the fixture on the
table. The good side of this is that it gives us something we care about: an
answer to the question, "How well does our program work?"
I'd like to move to a different question:
What can we know about our tests if we don't
understand the
fixtures?
Alternative Semantics
A different interpretation of the directory containing my test above might
yield this table as a result:
This is a cross-reference chart, showing where every fixture is used. I can
get this information with no reference to the fixture implementation
at all: it's just an
index of the first cell of each table.
We can find out other interesting things with just a little knowledge of
fixtures. For example, if we know that "Calculator" above is
a ColumnFixture, then we
know that the first row consists of input and outputs. Even without knowing
anything about how
the fixture is implemented, we can create another interesting
table:
| Vocabulary |
| Fixture |
Field |
Value |
| Calculator |
plus() |
1 |
| Calculator |
plus() |
3 |
| Calculator |
times() |
-2 |
| Calculator |
times() |
0 |
| Calculator |
times() |
2 |
| Calculator |
x |
0 |
| Calculator |
x |
2 |
| Calculator |
y |
-1 |
| Calculator |
y |
1 |
| Calculator |
y |
3 |
This table gives you a picture of the input and output domains. A
good tester
could look at this and notice all kinds of things that weren't tested:
- no large or small numbers (in either the input or output)
- no non-numbers
- no sums that resulted in 0
- no sums that were negative
- no sums that were even
- only even numbers for x, odd numbers for y
- minus(), divide()
- etc.
We could create a tool that would give us a domain analysis of our test
data:
| Test Data for Calculator |
| Field |
Max Neg |
Neg |
Zero |
Pos |
Even |
Odd |
Max Pos |
| plus() |
no |
no |
no |
yes |
no |
yes |
no |
| times() |
no |
yes |
yes |
yes |
yes |
no |
no |
| x |
no |
no |
yes |
yes |
yes |
no |
no |
| y |
no |
yes |
no |
yes |
no |
yes |
no |
Reporters and Rewriters
Two types of interpretations stand out for me:
Reporters - tell you something about a set of tests
Rewriters - change a set of tests
The Index and Vocabulary examples above are reporters. The standard Fit
interpretation is close to being a rewriter: it produces a version of the input
file, modified to show the results of the tests. (The only thing that keeps it
from being a true rewriter is that it leaves the original file in place, so you
can run it again.)
Here are some more ideas for useful tools:
- Count test cases- A reporter telling the total number of test cases for a
fixture
- Count fixture - A reporter telling how many times a fixture occurs
- Operators - A reporter telling the column names in RowFixtures and
ColumnFixtures (or the second column of ActionFixtures)
- Operands - A reporter telling the data values used (like
"Vocabulary"
above)
- Column changer - A rewriter that can rename, delete, or insert a
column
- Cell rewriter - A rewriter that can change cell values (for a specific
fixture in a specific column or row)
(I've created a simple version of the Index example above using the AllFiles
fixture; the others are speculation.)
Reflection is OK
There are useful semantics that don't try to interpret a fixture,
but it can help to "peek" a little. For example,
knowing that something is a
ColumnFixture tells us that it's likely that the row after the
fixture name consists of input and output fields. We can use this information
fruitfully. The Vocabulary example above made use of this knowledge.
Furthermore, there is nothing wrong with getting help. If someone had a
new type of fixture that subclassed Fixture, but still had ColumnFixture-like
semantics, they could provide a helper analysis class that would let us know
this.
The goal is not to avoid using fixture-aware code, it's just to avoid the
quagmire of trying to interpret
another program.
Call to Action
We've had a few years to work with Fit. People are creating test
suites large
enough to be interesting, and large enough that they need help
managing them.
It's time to experiment with new interpretations of Fit tests. (We still
may use Fit to help with this task.) The need is there now, by real people doing
real work.
[June, 2005.]
|