| Agile '06 Workshop: Example-Based Specifications |
This workshop was designed to explore Example-Based
Specifications. Hosted by Brian Marick and Bill Wake.
Schedule:
- 1 hour - One person plays customer, the rest of the team writes
tests
- 30 minute debrief
- 1 hour - Small group discussions
- 30 minutes - sharing and conclusions
Test Writing
Group 1: Domain - a statistical tool that evaluates
inputs according to a formula, and plots the results.
Their test:
(A, B, and Y are inputs; b0, b1, b2, and b12 are outputs,
produced by a process of matrix algebra.)
| A |
B |
Y |
b0 |
b1 |
b2 |
b12 |
| 10 |
75 |
8 |
11.0 |
5.0 |
8.0 |
12.0 |
| 20 |
75 |
12 |
| 10 |
100 |
10 |
| 20 |
100 |
15 |
This is a textual representation of the graph
| b0 |
b1 |
b2 |
b12 |
B-1 |
B-2 |
B+1 |
B+2 |
| 0 |
5 |
8 |
12 |
(1,1,RGB) |
(10,10,RGB) |
(10,1,RGB) |
(1,10,RGB) |
Their expected graph (described here, though the team drew a
picture):
- a large X with each line in a different colors
- labels +1 and -1 in the graph
- a horizontal axis labeled "A" (also having labels -1 to
+1)
- a vertical axis ("Y")
- a graph name ("B").
Group 2: Domain - a session scheduler, matching audiences
to rooms
Test: Schedule a session -
Preconditions: user has logged in, room size < audience
size
- TU35 has iaudience 100
- Room NICA has capacity 25
- TU35 is not scheduled
- NICA is available Monday 11-12:30
Execution:
- User assigns TU35 to NICA at Monday 11-12:30
- System displays a warning, "Room size mismatch... Continue or
cancel?"
- User selects "Continue"
Expected Results:
- TU35 shows up scheduled in NICA, Monday 11-12:30
Group 3: Domain - a graphical tool for tracking and
identifying criminal conspiracies. (Tool helps build a network of
connections between people).
Their test:
| Select template
criminal |
| Create node |
| Create node |
| Select |
link template |
? |
| Create link |
| Check |
count links |
1 |
| Delete node |
1 |
| Check |
count links |
0 |
| Create node |
policeman |
| Set |
first name |
Tom |
| Set |
last name |
Nassau |
| Create criminal |
| Set |
last name |
Soze |
| Select link template |
arrest |
| Create link |
Nassau to Soze |
| Check |
link exists |
| Select template |
policeman |
| Create node |
| Add role |
| Select node |
1 |
| Check |
has role |
policeman |
| Check |
has role |
drug dealer |
Group 4: Domain - an expense approval manager
A GUI mockup:
Manager approval:
Purpose:
| Type |
Date |
Description |
Amount |
Needs Receipt |
| |
|
American Airlines SEA to MSP |
$600.00 |
|
| |
|
|
|
X
|
| |
|
|
|
|
Issues: editing, are you sure?, manager approval, expense
group
Test:
| Type |
Date |
Vendor v |
Description |
Amount |
| Air tix |
7/16/06 |
AA |
ORD to MSP |
763.42 |
Purpose: Trip to agile conference
Manager approval:
Pointy-haired boss
An analysis of states:
New - Open, can't pay, can't approve, can't close
Submitted - Open, can't pay, can approve, can't close
| |
Is Open |
Can Pay |
Can Approve |
Can Close |
| New |
|
|
|
|
| Submitted |
|
|
|
|
| Approved |
|
|
|
|
| Paid |
|
|
|
|
| Closed |
|
|
|
|
Group 5: A stock trading program
A screen mockup with annotations:
| Start Time |
[ ] v : [ ] v |
9:30-16:00 ET (market start to market end) |
| End time |
[ ] v : [ ] v |
| Stock Symbol |
[ ] |
1-4 alpha |
| # Shares |
[ ] |
1 |
| Order Size |
[ ] |
100->1 million (int) (+- 100) |
| |
[Buy/Sell] |
|
| Price |
$ [ ] |
(optional) numeric + 2 decimal optional |
| |
|
|
| |
[OK] [Cancel] |
|
Test of algorithm:
- Start time >= now >= 9:30 EST
- End time > start time <= 16:00 EST
- Distribution list (file) exists
- Test invalid symbol entry
- Test for valid symbol entry
- Test all shares are traded if no limit price
- Test trades don't violate limit price
- Buys or sells based on selection
Group 6: Insurance tracking program
Test for story "Insurer adds provider":
Test 1 - Sunny day
Add table:
| Johnson, David |
name |
15ch, 15ch |
| 1200 Nicolette Dr |
addr1 |
40ch |
| 12345 |
addr2 |
5d or 5d-4d |
| 123456789 |
taxid |
9 ch num |
| ALLINA |
network id |
must exist in db |
Read: (same)
Test 2- Invalid zip code (and more like this...)
| Johnson, David |
name |
15ch, 15ch |
| 1200 Nicolette Dr |
addr1 |
40ch |
| 1234 |
addr2 |
5d or 5d-4d |
| 123456789 |
taxid |
9 ch num |
| ALLINA |
network id |
must exist in db |
Expected error: "Invalid zip could should be 5 digits"
Test for story "Analyst approves pending claims < 60 days
since claim made"
Test 1 - Sunny day
Populate database - set up claim 1
| Date of claim |
30 days ago |
| Member name |
Doe, Jane |
| Provider name |
Johnson, David |
| Diagnosis |
769 |
| Charge |
$500.00 |
| Network id |
ALLINA |
Step 1. Analyst views pending claims < 60 days (display claim
A => OK)
Step 2. Analyst approves pending...
Signal "ok" = submit
Claim disappears from list
Message "approved"
Step 3. Read claim, status "approved"
Group 7: Domain: Shipping company.
Test: 1. Customer ABC call to know where shipment with order #33
is. The system should answer, "Last stop was Tampa, FL."
| Note |
Order # from Customer |
Request |
Answer from System |
| Truck left origin |
#33 |
last stop? |
Tampa, FL |
| Not in truck |
#34 |
last stop? |
nothing |
| Truck at destination |
#35 |
last stop? |
Daytona, FL |
| Truck at destination |
#35 |
arrived? |
yes |
| Not in truck |
#34 |
arrived? |
no |
| Not arrived |
#34 |
expected date? |
26/7 |
| Arrived |
#35 |
expected date? |
25/7 |
| Truck underway |
#33 |
arrived? |
no |
Example context:
| # |
Pick up origin |
Final drop destination |
Expected Date |
| #33 |
Tampa, FL |
Toronto, ONT |
27/7 |
| time 24/7 8 AM |
|
| #34 |
Vancouver, BC |
Toronto, ONT |
26/7 |
| time |
|
| #35 |
Toronto, ONT |
Daytona, FL |
25/7 |
| time 24/7 17:00 |
time 25/7 10:00 |
Test 2: I want the system to help me minimize empty truck
displacement. For example, I want to be able to ask if there is an
empty truck in Ontario on July 27. Arrival within 2 days.
| Empty truck in? |
Shipment order |
|
| Ontario/27/07 |
#33, #34 |
truck at city |
| Ontario/25/07 |
|
no truck because wrong date |
| Montreal/QC |
|
no truck because wrong location |
Debrief
We noticed these things from the experience of writing
tests:
- What to do is vague.
- Developers tend to embellish.
- Tests teach developers, but it's a challenge to pick the right
level.
- It's hard to turn descriptions into tests.
- Tests (and collaboration) can help you discover new
things.
- The idea of a GUI became actions on a model (for the
test).
- Customers should come in prepared.
- We need many questioners per answerer.
- Someone came in late, and found that the examples helped them
understand.
Topics
In small groups, we explored these topics:
- Sufficient coverage: How do you know you're done - what's
sufficient coverage?
- Test styles: What is the relationship between example-based
specifications and other styles of tests?
- Product Owner: Techniques for interviewing product owners
Sufficient Coverage
- When adding tests to a legacy system, how much do you
"backfill"?
- Should we just use "change detector" tests (record what the
system currently does, knowing that it may change later, and may
not even be correct).
- Do we need all combinations?
- Where do we fit example-based tests?
Test Styles
We suspected these things about example-based tests:
- They may bring about exploratory testing, regression tests,
better unit tests, acceptance tests. Load tests?
- They provide a concise account of an edge case.
- They serve to train new developers in a domain.
- They provoke a certain style of conversation.
- They may overwhelm developers with distracting detail (no
"metaphor").
- Alternatively, developers may ignore examples! It
happens...
Product Owner
- Use Sophie's Rule - keep asking "why?"
- Ask the end goal, define the business problem, define
acceptance criteria
- Discuss requirements "as if you were blind" (without reference
to UI)
- Need a customer who is is readily accessible
- Helps to talk to end users
- Programmers need to understand the domain
- Let product owners ramble
General Notes / What Next
- Mind maps have utility in these conversations.
- Examples are a style of conversation - the easiest kind to
get.
- "We're going to practice" - "process miniature" exercises could
help.
- A whiteboard and pointing are powerful ways to focus
attention.
|