XPlorations
| XP123 → XPlorations → Procedural and Declarative Tests (March, 2005) |
| Procedural and Declarative Tests | March, 2005 |
| Procedural tests focus on a series of steps; declarative tests work from the input and states. Procedural tests are natural to write, but consider whether a declarative test expresses your intent more concisely and clearly. |
Test Styles
There are two common styles of writing tests. One style might be called procedural: a test consists of a series of steps, each acting on or testing the state of the system. The second style might be called declarative: given a state of the system and some inputs, test the resulting state and the outputs. (These terms are from the theory of programming languages.) If we were using Ward Cunningham's fit testing framework, we might think of these as prototypically ActionFixtures vs. ColumnFixtures.
Let's look at a small, concrete example: a simple login screen.
We'd like to test that the OK button is highlighted only if both a username and a password have been specified.
Here's a test in the procedural style:
| 1 | enter | username | bob |
| 2 | expect | OK | inactive |
| 3 | enter | password | pw |
| 4 | expect | OK | active |
| 5 | clear | username | |
| 6 | expect | OK | inactive |
| 7 | clear | password | |
| 8 | expect | OK | inactive |
Here's a test of the same capability in a declarative style:
| Username | Password | OK active? | |
| 1 | none | none | no |
| 2 | bob | none | no |
| 3 | none | pw | no |
| 4 | bob | pw | yes |
There's a sense in which the first test is more natural to write: it tells you what to do, step by step. But consider some advantages of the second style:
- If we want to know whether a particular test case is covered, the second style shows it more directly. For example, what if the username and password are both empty? It's easy to see that this case has been considered in the second test.
- The declarative style fails or passes one row at a time. The procedural style is vulnerable, in that if a check in the middle fails, the whole rest of the test may be invalid. (This isn't "free" though - the code connecting the test to the system must be designed to make this test independence work.)
- The declarative style has all the critical state, inputs, and outputs listed explicitly. In the procedural style, you may have to trace the state through many steps.
- The procedural style tends to presume it knows details about the interface; this makes it more brittle to change.
Keeping State
Real systems may involve hidden state: state that affects the system but is not directly set or seen. Consider this example:
This is an accumulator: it provides a running sum and average. Obviously, it must somehow keep track of n, the number of items entered, in order to compute the average.
We can make a declarative test out of this by exposing the hidden state to the test:
| n | sum | data | sum' | avg' | comment |
| 1 | 100 | 100 | 200 | 100 | normal |
| 9 | 100 | 0 | 100 | 10 | add 0 |
| 2 | 2 | 0 | 2 | 0 | integer divide |
The first two columns, n and sum, represent existing state. Data is the number entered on the screen, and sum' and avg' are the results.
Getting to the hidden state can be a trick. Sometimes the programmers can explicitly expose it as part of a testing interface; other times the state can be accessed by a sequence of steps. In this case, we could setup the proper state for each row by doing this:
Hit the clear button
Repeat n-1 times: add 0
Add sum
[Optional: check sum]
Then the rest of the row might be interpreted as:
Add data
Check sum
Check avg
This puts some burden on the test setup code. But if the setup is not done there, it's probably repeated in a bunch of scripts.
When to Use the Procedural Style?
Procedural style pays off best when the sequential nature of the feature is what's interesting: when how you got somewhere is as important as where you are. Consider the problem of multiple selection, using click, shift, control, and drag. For setup, imagine a vertical list of items, numbered 1 through 10. Consider this test:
| 1 | click | 5 | |
| 2 | release | ||
| 3 | expect | selected | 5 |
| 4 | expect | last | 5 |
Or this one:
| 1 | click | 5 | |
| 2 | drag to | 7 | |
| 3 | drag to | 3 | |
| 4 | release | ||
| 5 | control | ||
| 6 | click | 7 | |
| 7 | expect | selected | 3,4,5,7 |
| 8 | expect | last | 7 |
You could perhaps develop a set of declarative tests for these, but the sequence of actions is what's interesting.
What to do?
Leverage procedural tests for the cases where the sequence of actions is paramount, and there's not a lot of commonality between scripts. Scenario tests can benefit from the procedural style.
A declarative test is like a table you could put in a user manual: it concisely and concretely explains a function. Declarative tests sometimes require extra setup, especially when hidden state is involved, but they're often worth that extra trouble. Declarative tests are good for testing permutations of a business rule.
If all else is equal, favor the declarative style.
These guidelines can boost your test-writing efficiency: they move repetitive actions into the test setup, and let you focus on the interesting part of a test.
[Written February, 2005. Brian Marick suggested the terms, though I'm not sure I picked the set he liked best.]