Requirements, acceptance testing and BDD/ATDD

1. Requirements and stakeholders

Chapter 1 argued that robust software engineering is about controlling risk with process and with automation. This chapter takes a step back and asks the question that comes before any test: what exactly is the software supposed to do, who decides that, and how do we write it down so that a machine can check it. The course answers with a very compact definition.

That immediately raises the question of whose wants count, and the deck defines the second term just as tightly. A stakeholder is someone, or something, that derives value, benefits from, or influences our system. The word "something" is deliberate: a stakeholder need not be a human being. Another piece of software that consumes our API, or a regulator whose rules constrain us, influences the system just as much as a person does.

Stakeholders come in two flavours, and the distinction matters when you decide whom to interview and whom to demo to:

• Non-acting stakeholders: directors, business owners, enterprise architects, clients. They do not touch the running system, but they pay for it, own it, or constrain its shape. They are the source of business requirements.
• Acting stakeholders: UI users, API users, and the actors that the system needs to be assisted by. These are the ones who will actually sit in front of the interface or issue the calls, and also the external actors and services our system depends on to do its job.

This is more than terminology. If developers were stakeholders, the requirement set would quietly absorb implementation preferences, and acceptance tests would end up testing the code we wrote rather than the value we promised. Keeping the boundary sharp is what makes the rest of the chapter, and acceptance testing in particular, possible at all.

2. Kinds of requirements, tracking, and what makes one good

Kinds by role

Requirements are classified by the role they play in the project, and the deck lists five roles. They form a rough ladder from business intent down to technical constraint:

Ways of tracking requirements

The second axis is representation: how a requirement is written down and kept. The deck lists formal statements, business rules, examples, user stories, screen mock-ups, narratives and diagrams. These are not competing schools, they are different registers of the same content, and real projects mix them. A screen mock-up conveys a layout requirement in one second and would take three paragraphs of prose; a business rule is precise where a narrative would be vague. The important observation for this chapter is that examples are on the list: an example is a concrete instance of desired behaviour, and an example is exactly the kind of requirement that a machine can execute. That is the crack through which Gherkin and Cucumber will enter.

The seven properties of a good requirement

Whatever the kind and whatever the notation, the deck gives seven quality properties. A good requirement is:

1. Unambiguous: it admits one reading, not several.
2. Understandable: the stakeholder who expressed it can still read it back and recognise it.
3. Testable: there is a procedure, at least in principle, that decides whether it holds.
4. Feasible: it can actually be built, with the technology and the budget available.
5. Implementation-free: it says what, not how. It does not smuggle in a design decision.
6. Consistent: it does not contradict other requirements in the set.
7. Complete: nothing essential is left implicit, and the set as a whole covers the need.

Two of these are in permanent tension with each other, and noticing that is worth an exam point. Unambiguous pushes towards formality, while understandable pushes towards the stakeholder's own language. Gherkin, later in this chapter, is essentially an engineering compromise between the two: enough structure for a tool to parse, enough prose for a business analyst to sign off on.

Use the widget below to practise the judgement. Each candidate is a plausible-looking requirement; click it and see which of the seven properties it breaks.

3. From requirement to specification

A requirement on its own is not very useful. The deck is blunt about it: requirements are little useful per se, they should rather be given context, scope, and precision, and this is achieved by turning them into specifications. Context tells you when the requirement applies, scope tells you how far it reaches, and precision tells you what counts as satisfying it.

The step from requirement to specification is where behaviour-driven development enters. The course states its aim in one line: BDD aims at turning requirements into executable specifications. "Executable" is the operative word. A specification written in prose has to be read by a human before anything can be checked; a specification written so that a tool can run it becomes a regression test that never goes stale, because the moment the code drifts away from it the build turns red.

The chain does not stop at specification. Conceptually, specifications can be turned into design, and design can be turned into implementation, with high guarantees that the implementation conforms to the requirements. The word "conceptually" is doing real work there: in ordinary practice these transformations are performed by hand, by people, and the guarantee is only as good as their discipline. The course notes that this chain becomes practical in so-called Model-Driven Engineering (MDE), where the transformations are mechanised and the models, rather than the code, become the primary artifacts. MDE returns later in the course as a topic in its own right; here it is enough to see where it sits on the chain.

4. The requirements life cycle

Requirements are not written once and then obeyed. They have a life cycle, and the deck presents it as a diagram rather than as a bulleted list. The loop it depicts is the standard one of requirements engineering, and it is worth spelling out because every stage of it will reappear when we look at the ATDD cycle.

Requirements are first elicited from stakeholders, by interviews, observation or examples. They are then analysed, which is where contradictions and gaps surface and where the seven quality properties get applied. They are specified, that is, turned into descriptions of required behaviour with context, scope and precision. They are validated with the stakeholders, who confirm that what was written is what they meant. And then they are managed: the world changes, the business changes, and requirements are added, retired and revised, feeding the loop again.

5. Acceptance testing

With requirements and specifications in place we can define the kind of testing that judges them. The deck's definition:

Read the definition carefully, because two of its choices are unusual. It says "to what degree", not "whether": acceptance is a matter of degree, and a system can be accepted with reservations. And it says "end users' approval", not "correctness": the verdict belongs to people, not to an oracle. That is the difference in kind between acceptance testing and the unit testing of Chapter 1.

The deck attaches four characteristics, and they define the shape of everything that follows:

1. Very often it is not automated, but involves physical interaction with stakeholders. Someone sits down in front of the system and tries it. Much of this chapter is about how far we can push against this default, and the honest answer, in section 13, is: partly.
2. It is a case of black box testing, where we do not care about the API, but just about "results". Nobody is looking at internal classes, and no test asserts on a private field.
3. Typically it is done after integration and system testing, and before production. It is the last gate. By the time acceptance testing starts, the parts already work together; what is being asked is whether the working whole is the right whole.
4. Sometimes it is an essential element of software agreements and contracts. When a supplier delivers to a client, the acceptance test suite can literally be the definition of "delivered", with money attached to it.

6. ATDD: acceptance test-driven development

Recall TDD from the previous chapter: in the TDD technique, a single unit test is written before writing the (smallest) production code that passes the test. The rhythm is red, green, refactor, and the unit of progress is one small test.

ATDD keeps the rhythm and lifts the altitude:

The crucial words are "with TDD". ATDD does not replace TDD, it wraps it. The outer loop picks one acceptance test and holds it red; the inner loop is ordinary TDD, one unit test at a time, until the outer test turns green. You end up with two nested feedback loops running at two different speeds: seconds for the unit tests, minutes or a demo cycle for the acceptance test.

The deck lists three differences between the two techniques, and they line up neatly with the definitions from section 1:

The third difference is the deepest one and it is what motivates Gherkin. A TDD test is written in the language of the API, which is fine because its author owns the API. An acceptance test written in the language of the API silently violates the black box characteristic of section 5, and locks a business promise to a method signature. Section 9 shows exactly that failure mode in code.

7. BDD, Given-When-Then, and the ATDD cycle

Behaviour-driven development is defined by the deck as follows: BDD is a testing approach where tests are mainly based on systems' behaviour, typically expressed by the Given-When-Then approach. The triple is the whole grammar of the idea:

• Given establishes the context: the state of the world before anything happens.
• When is the event: the single action whose consequences we care about.
• Then is the expected outcome, observable from outside.

This is the classical arrange, act, assert of unit testing, restated in words a non-programmer can use. That restatement is the entire contribution: it moves the test from the developer's vocabulary into the stakeholder's.

How do BDD and ATDD relate? The deck is explicit that they are very similar, and essentially we shall use them as synonyms, while noting a difference of emphasis:

• ATDD is more seen as an extension of TDD, to deal with acceptance tests written from actual requirements.
• BDD is more seen as a black-box starting point for system development, not necessarily for acceptance.

In other words, ATDD comes at the idea from the testing side (start from TDD, raise the level) and BDD from the specification side (start from behaviour, then build). They meet in the middle, which is why the same tool, Cucumber, serves both.

The ATDD cycle itself is shown in the deck as a diagram. Explore its stages below, and notice how the inner TDD loop nests inside it.

8. Goals, capabilities and features

Before writing a single scenario, it helps to know at which altitude a scenario is supposed to fly. The deck gives three levels, each with a one-word question attached:

The three levels form a chain of justification. A goal explains why anyone should care; a capability says what the user will be able to do about it; a feature says what the system must actually provide so that the capability exists. Take the calculator example that runs through the deck: the goal is "I do not want to learn maths", the capability is "I want to be able to add numbers using a Calculator", and the features are the individual behaviours (adding two positive numbers, adding a positive and a negative, and so on).

9. Motivation: an acceptance test written in JUnit

Suppose we simply write the acceptance test with the tools we already have. The deck's motivating example is a calculator, tested with plain JUnit:

public class CalculatorTest {

    @Test
    void acceptanceTest() {
        Calculator calculator = new Calculator();
        calculator.enter(5);
        assertEquals(5, calculator.getResult());

        calculator.enter(6);
        calculator.add();
        assertEquals(11, calculator.getResult());

        calculator.enter(3);
        calculator.add();
        assertEquals(14, calculator.getResult());
        assertEquals(14, calculator.getResult());

        calculator.enter(3);
        assertThrows(IllegalStateException.class, () -> calculator.enter(5));
    }
}

There is nothing wrong with this test as a test. It is readable, it covers a realistic session, and it even pins down an error case (entering a third operand without an operator throws). What is wrong with it is its role. The deck states the point directly: BDD/ATDD is not about APIs. Acceptance testing starts from requirements defined according to business analysts and domain experts, and without specific technologies those requirements tend to resemble system tests and are API-dependent.

Look at what a stakeholder would have to accept in order to sign off on this file. They would have to accept the names enter, add and getResult, the fact that operands are pushed one at a time before the operator (a stack-machine convention that nobody asked for), and the choice of IllegalStateException as the failure signal. None of that is a requirement. All of it is design. The requirement underneath is much smaller and much more stable: adding 5 and 6 gives 11.

10. The Gherkin language

Gherkin is the answer to the previous section. The deck introduces it as a simple specification language for features, part of Cucumber, and characterises its flavour precisely: it is not programming, but structuring user stories. The reference lives at cucumber.io/docs/gherkin/reference/. Crucially, it can be used to generate template executable tests, called "steps". That single sentence is the bridge between prose and code: you write the story, the tool writes the empty test methods, and you fill them in.

The main syntactic elements are few:

Notice that Scenario is glossed as "an example". This is the deck's way of saying that Gherkin is an example-based notation: you do not state a universal law, you exhibit concrete cases, exactly the "examples" listed among the ways of tracking requirements back in section 2. Here is the deck's full calculator specification:

Feature: Adding numbers with a Calculator
  In order to not learn math
  As someone who is bad at math
  I want to be able to add numbers using a Calculator

  Scenario: Add two positive numbers
    Given I have a Calculator
    When I add 1 and 1
    Then the sum should be 2

  Scenario: Add a positive and negative number
    Given I have a Calculator
    When I add 1 and -1
    Then the sum should be 0

  Scenario: Add two negative numbers
    Given I have a Calculator
    When I add -1 and -1
    Then the sum should be -2

The three lines under Feature are free prose, ignored by the parser, and they map exactly onto the three levels of section 8: "in order to not learn math" is the goal, "I want to be able to add numbers using a Calculator" is the capability, and each Scenario below is a feature. Compare this file with the JUnit test of section 9: no class name, no method name, no exception type, and yet the same behaviour is pinned down. That is what "not about APIs" buys you.

Build a scenario, and see the steps it would generate

Pick one Given, one When and one Then below. The widget assembles the scenario and shows the step definitions Cucumber would generate from it, including how literal values in the prose become typed parameters in the signature.

11. Cucumber and the IntelliJ/SBT/Java toolchain

Gherkin describes; Cucumber executes. As the lab deck puts it in one line, Cucumber binds Gherkin to code execution, of the "steps". The deck lists the ingredients needed to get this running in IntelliJ with SBT and Java:

• Plugins for the IDE: Gherkin, and Cucumber for Java.
• A dependency for the build tool: "io.cucumber" % "cucumber-java" % "6.1.1" % Test.
• The feature file in a resource folder of the module: test/resources/features/*.feature.
• Generation of the step code: ask the IDE from the contextual menu of a step.
• Execution: play the tests directly on the feature files.

Two details are worth pausing on. The dependency is scoped % Test, so Cucumber never ships with the production artifact: the executable specification is test infrastructure, not a runtime library. And the feature files live under resources, not under sources, because they are data to be parsed, not code to be compiled. That is also why a stakeholder can edit one without touching a compiler.

Generated steps

Asking the IDE to generate steps for the calculator feature produces a skeleton in which each distinct step text has become an annotated method:

package calculator;

import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;

public class CalculatorSteps {

    @Given("I have a Calculator")
    public void iHaveACalculator() {
    }

    @When("I add {int} and {int}")
    public void iAddAnd(int arg0, int arg1) {
    }

    @Then("the sum should be {int}")
    public void theSumShouldBe(int arg0) {
    }
}

Three scenarios produced three methods, not nine, because the three scenarios reuse the same three step texts with different numbers. Where the prose said 1, -1 or 2, the generated annotation says {int}: Cucumber recognised the literal as a typed parameter and turned it into a capture group, delivered to the method as an int. This is the whole binding mechanism, and it is why one step definition can serve an unbounded family of examples.

Filling the skeleton with code

Click any line of the deck's filled-in version to read what it does.

Notice the two decisions the deck makes explicit in this class. The When step only enters the operands, while the Then step performs the add() and compares: the arithmetic is triggered where the outcome is checked, which keeps each step small but does mean the mapping between Gherkin steps and API calls is a design choice, not a mechanical translation. And the failure is signalled by throwing IllegalStateException, with the deck's own comment noting that one could use JUnit's asserts instead. Cucumber declares a step failed when it throws, so any assertion library works.

12. The rest of Gherkin: Background, Scenario Outline, Doc Strings, Step Tables

The three-scenario calculator file repeats Given I have a Calculator three times, and it spells out three near-identical scenarios that differ only in two numbers. Gherkin has constructs for both problems, plus two ways of carrying richer data into a step. Each tab below shows the deck's own example.

Feature: Adding numbers with a Calculator
  In order to not learn math
  As someone who is bad at math
  I want to be able to add numbers using a Calculator

  Scenario: Add two positive numbers
    Given I have a Calculator
    When I add 1 and 1
    Then the sum should be 2

Feature: Adding numbers with a Calculator
  In order to not learn math
  As someone who is bad at math
  I want to be able to add numbers using a Calculator

  Background: Start with a Calculator
    Given I have a Calculator

  Scenario: Add two positive numbers
    When I add 1 and 1
    Then the sum should be 2

  Scenario: Add a positive and negative number
    When I add 1 and -1
    Then the sum should be 0

  Scenario: Add two negative numbers
    When I add -1 and -1
    Then the sum should be -2

Feature: Adding numbers with a Calculator
  In order to not learn math
  As someone who is bad at math
  I want to be able to add numbers using a Calculator

  Scenario Outline: Add two numbers
    Given I have a Calculator
    When I add <arg0> and <arg1>
    Then the sum should be <res>
    Examples:
      | arg0 | arg1 | res |
      | 1    | 1    | 2   |
      | 1    | -1   | 0   |
      | -5   | -6   | -11 |

Feature: Google Searching
  As a web surfer, I want to search Google, so that I can learn new things.

  Scenario: Simple Google search
    Given a web browser is on the Google page
    When the search phrase "panda" is entered
    Then results for "panda" are shown
    And the result page displays the text
      """
      Scientific name: Ailuropoda melanoleuca
      Conservation status: Endangered (Population decreasing)
      """

Feature: Google Searching
  As a web surfer, I want to search Google, so that I can learn new things.

  Scenario: Simple Google search
    Given a web browser is on the Google page
    When the search phrase "panda" is entered
    Then results for "panda" are shown
    And the following related results are shown
      | related       |
      | Panda Express |
      | giant panda   |
      | panda videos  |

Step tables at full stretch: the chess example

The deck closes its Gherkin tour with an example that shows how far a step table can be pushed. A whole chess position is passed to a When step as a table, and the expected king moves are checked against another table:

Feature: King movement

Scenario: A king in its beginning position
  When there is a chess board set up as
  | x | a  | b  | c  | d  | e  | f  | g  | h  |
  | 1 |    |    |    | BK |    |    |    |    |
  | 2 |    |    |    |    |    |    |    |    |
  | 3 |    |    |    |    |    |    |    |    |
  | 4 |    |    |    |    |    |    |    |    |
  | 5 |    |    |    |    |    |    |    |    |
  | 6 |    |    |    |    |    |    |    |    |
  | 7 | WP | WP | WP | WP | WP | WP | WP | WP |
  | 8 | WR | WN | WB | WQ | WK | WB | WN | WR |
  Then the WK at E8 should have the following moves
  | Start | End |

Scenario: The pawn was moved up
  When there is a chess board set up as
  | x | a  | b  | c  | d  | e  | f  | g  | h  |
  | 1 |    |    |    | BK |    |    |    |    |
  | 2 |    |    |    |    |    |    |    |    |
  | 3 |    |    |    |    |    |    |    |    |
  | 4 |    |    |    |    |    |    |    |    |
  | 5 |    |    |    |    |    |    |    |    |
  | 6 |    |    |    |    | WP |    |    |    |
  | 7 | WP | WP | WP | WP |    | WP | WP | WP |
  | 8 | WR | WN | WB | WQ | WK | WB | WN | WR |
  Then the WK at E8 should have the following moves
  | Start | End |
  | E8    | E7  |

The first scenario expects an empty result table: with every square around it occupied by its own pieces, the king cannot move at all. In the second, the e-pawn has advanced, one square opens up, and the expected moves table has exactly one row. The example is instructive because it shows Gherkin used pictorially: the table is not decoration, it is the argument, and a domain expert who knows chess can check the specification without knowing anything about the board representation chosen in code.

13. Automating acceptance tests

Section 5 warned that acceptance testing very often is not automated. Now that we have Cucumber, how much of that gap can we actually close? The deck is careful, and the answer has two halves:

• Some acceptance tests can be fully automated, when considering functional aspects. The calculator is the easy case: input goes in, a result comes out, and the expected result is a number.
• Some can hardly be automated, especially when dealing with input/output aspects. The deck names two: usability aspects of the UI, and integration with external resources like databases or servers.

The two obstacles are of different natures, which is worth stressing. Usability is hard to automate because the property itself is subjective: no assertion decides whether a screen is confusing, only a user does. Integration with external resources is hard for a different reason: the property is objective, but the resource is outside our control, slow, stateful or simply absent from the build server.

For the second category the deck proposes concrete machinery: dependencies with external frameworks can be "managed" by suitable drivers, which create sorts of proxies facilitating the tracking of events for testing purposes. The examples given are AssertJ Swing for desktop GUIs and Selenium for the web. A driver of this kind sits between the test and the real component, exposes a programmable interface over something that was designed for a human, and lets the test observe events it could otherwise not see. The deck adds that mocking will be used as well, which is the subject of the next chapter: where a driver wraps the real thing, a mock replaces it.

14. Lab 01: Cucumber, ATDD and LLMs

How labs work in this course

Each module of ASMD has a lab activity, and each lab proposes two classes of exercises:

• Operational steps, designed as quick exercises to be completed during the lab session itself.
• R&D tasks, more in-depth "research and development" activities, similar to exam projects, which typically require additional work outside the lab.

The R&D tasks are not meant to be all completed: pick one, or two, or more, depending on your interest and the time you have, start in the lab and continue at home. Students work alone or in small groups, typically in pairs. The teachers provide assistance as needed, during the lab or later offline, and ask that you share interesting results or completed tasks as soon as you have them: send an email with the subject [ASMD24-LABXX-TASKYY] and put authors and repository in the body.

On the exam: assume around 90 to 100 hours of work in addition to the labs. The exam is a discussion of the tasks completed by the student, for example one big task producing a software artifact (which can later become a scientific paper) or various smaller tasks, plus a discussion of the links to other parts of the course.

The one-slide sum-up

The lab deck compresses the whole module into three groups of lines, which are worth memorising as they stand:

• Requirements and specification: a requirement is a stakeholder's expression of a need or wish with regard to the system; a specification is a description of system behaviour required to fulfil a requirement.
• ATDD/BDD: acceptance testing is testing on the overall system to determine approval by stakeholders; ATDD means determine or pick an acceptance test, then use TDD to make it pass; BDD is essentially ATDD, using behaviour-oriented tests instead of acceptance tests; in ATDD/BDD, use tests expressing "features".
• Gherkin/Cucumber: Gherkin is a language to express features, mixing prose and keywords; Cucumber binds Gherkin to code execution, of the steps.

References and goals

The lab points at cucumber.io for Cucumber, at the Gherkin reference for the language, and at the starting repository github.com/mviroli/asmd23-public-01-atdd. It also lists LLM front-ends to experiment with (ChatGPT, Perplexity, Claude, Gemini) and notes that slide 01b, that is, the material of sections 10 to 12 above, provides rather complete examples of Gherkin syntax. The general goals are: be operative with Cucumber, JUnit, Java and IntelliJ; exercise writing requirements with Gherkin; exercise ATDD; and pre-check the ability of LLMs to deal with Gherkin, which the deck flags as still very preliminary and something the course will deepen later.

Operational steps

Step 1, get ready. Clone the repository 01-repo-atdd, open it in IntelliJ, install the Cucumber plugin (both the Cucumber for Scala and the Cucumber for Java plugins), and run the tests: both the JUnit ones and the Cucumber ones. For Cucumber, go to src/test/resources and run the feature files in the features directory.

Step 2, play with Cucumber and ATDD. Add operations to the calculator, for example multiplication; write Gherkin specifications for the new operations; implement them; and repeat the process for other operations. The deck insists on the order in capitals: follow the ATDD process, that is, specification first, implementation after. Then ask yourself the deck's closing question: does everything work as expected?

R&D tasks

Test your knowledge