Advanced testing: TDD, test doubles and integration

1. Three definitions of software testing

The module opens by refusing to give a single definition, and offering three instead. They are not competing: each one is what testing looks like from a different vantage point, and each one licenses a different set of techniques.

Read them side by side and you get the shape of the chapter. The stakeholder view is why chapter 2 exists at all. The simulation view is what a JUnit test literally does, and it is also, word for word, what a Gillespie run does in chapter 12: execute under specified conditions, record, evaluate. The process view is the one that includes static activities, which is the licence for treating a model as a testable artefact.

2. The five aspects and the five schools

Before any technique, the deck lists the five questions any testing decision answers: why you test, that is, what the goal is; what you test, the subject or system under test; who prepares tests, runs them and evaluates results, and the relationships with other stakeholders such as developers and customers; when tests are prepared and executed with respect to when the system under test is built, before, after or altogether; and how testing is done, meaning which techniques and tools.

Different professional communities answer those questions differently, and the deck names five schools of testing for programmers. Learning them is worth the effort because each one gives a different answer to a single question: what is a test suite?

3. Levels of testing and the ISTQB vocabulary

Focusing on the what, the deck lists five levels:

1. Unit testing: testing at the level of individual units, of functionality or of code.
2. Integration testing: testing the functionality provided by multiple integrated or interacting units.
3. System testing: testing the system, whole or in part, for correctness, both for functional requirements in a black-box way and for non-functional ones such as efficiency, reliability, usability and security.
4. Stress testing: testing performance, memory and speed, in edge cases.
5. Acceptance testing: testing features against the expectations of users and stakeholders.

Alongside the levels comes the vocabulary, taken from the ISTQB glossary. It is worth memorising because the rest of the course reuses these words precisely.

The deck then folds all of it into one sentence, which is the best single-line summary of the vocabulary: you run a suite of test cases, using tools and oracles, to verify a set of test conditions on a set of testable test items.

4. Oracles, automation and specification

The oracle is the part of testing that automation cannot conjure out of nothing: something has to know what the right answer is. The deck enumerates the ways of getting one.

• Specified oracle: the results are known from the specification. Executable formal specifications, assertions, and so on.
• Derived oracle: using information from other artifacts, such as previous runs, reference systems, crash reports or regression tests.
• Implicit oracle: using implicit information, such as overflows, deadlocks, races or structural anomalies.
• Human: for example the usability of a UI, and generally acceptance testing.
• An LLM: used as judge of code.

And it closes the loop with model-driven engineering: specifications, and the architectural and design models derived from them, can be used as oracles to check the corresponding integration and system tests. That single line is the operational content of the framing diagram from chapter 1.

5. Test-driven development

TDD is presented as a new role for testing, promoted by the Agile Manifesto: testing moves from being the step next to coding, which is next to design to being the step in coding and design. The deck calls TDD the extreme which all modern practitioners naturally tend to.

The rationale is mechanical: instead of writing production code directly, you (i) write the minimal test that would use it, (ii) write the minimal code that makes it pass, and (iii) refactor and improve. Writing test code, writing production code and finding the optimal detailed design chase one another.

The benefits claimed are a high degree of control over how your production code behaves, high confidence in correctness and in changes, and a dramatic reduction of the time spent in debugging, summarised by the motto: debugging solves problems you created; TDD does the same faster and with added value. The drawbacks are honestly stated too: being 100 percent TDD requires high discipline and really a different mindset, and it requires programming skill, since with TDD refactoring affects both production and test code.

On writing good tests for TDD, three points are made up front. Software for tests is a system in its own right and should be technically excellent too: the system to build is made of several subsystems, namely production code, tests, configuration files, documentation, tutorials and data. Tests should be clean, simple, fast, independent and repeatable. And the technologies push for the idea of tests as readable sentences.

The running example: a Device

The whole module is built on one small example, deliberately trivial so that the testing technique stays visible.

public interface Device {
    boolean isOn();

    void switchOn();

    void switchOff();
}

public class DeviceImpl implements Device {
    private boolean on;

    @Override
    public boolean isOn() {
        return this.on;
    }

    @Override
    public void switchOn() {
        if (this.on) {
            throw new IllegalStateException();
        }
        this.on = true;
    }

    @Override
    public void switchOff() {
        this.on = false;
    }
}

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

public class DeviceTest {

    private Device device;

    @BeforeEach
    void init() {
        this.device = new DeviceImpl();
    }

    @Test
    void initiallyOff() {
        assertFalse(this.device.isOn());
    }

    @Test
    void canBeSwitchedOn() {
        this.device.switchOn();
        assertTrue(this.device.isOn());
    }

    @Test
    void canBeSwitchedOnAndOff() {
        this.device.switchOn();
        this.device.switchOff();
        assertFalse(this.device.isOn());
    }

    @Disabled
    @Test
    void cantSwitchOnIfAlreadyOn() {
        this.device.switchOn();
        assertThrows(IllegalStateException.class, () -> this.device.switchOn());
    }
}

Note the last test is annotated @Disabled: a test that describes behaviour not yet agreed on, parked rather than deleted. That is already a small piece of the specification role of a suite.

6. JUnit in practice

Annotations

Organising tests

The guidelines given are about the test suite as a designed artefact, not about syntax.

• Group by set of related features, not strictly by methods or classes.
• Put tests in the same package as the production code, to handle access control.
• Use inheritance: only non-abstract classes can be run, and all methods annotated @Test, inherited or not, will be executed. Hence common tests for diverse implementations can live in abstract test classes. The deck adds a principled justification: the Liskov substitution principle among the SOLID principles calls for replicating the subtyping of production code with subtyping in test classes.
• Use delegation to reuse test code, again mimicking the production code.
• Use nesting as needed: nested test classes, which need the @Nested tag, group test methods.
• Tag whenever it makes sense to run different subsets of tests, for example dividing unit, integration, system and stress tests, or by cross-cutting concerns.

Asserts, and assume

• assertEquals: recall the problem with precision error and doubles.
• assertNull, assertFalse, assertTrue, fail: be specific.
• assertThrows: use this instead of the expected parameter of @Test, if possible.
• assertAll: favour a single assertAll at the end of the test case.
• assertTimeout: be careful with timing in tests.

Assumptions (assumeTrue, assumeFalse, assumeEquals) are rarely used in practice, and their semantics is different in an important way: a wrong assumption is tracked differently, as not executed rather than as failing. The deck glosses it twice: a failed assumption does not mean the code is broken, but that the test provides no useful information; and, more bluntly, assume basically means do not run this test if these conditions do not apply.

7. How many unit tests, and which ones?

Unit testing helps maintain project growth, but the deck immediately qualifies it. It is not enough to just write tests: things get complicated because of integrations, refactoring, changes in requirements, legacy code, code that will not be tested, and so on. A project with badly written tests exhibits the properties of a project with good tests at the beginning, and eventually falls into a stagnation phase.

Two nuances are worth quoting. First, bad tests still seem to be better than no test: they are an instance of the technical debt issue, not of pure loss. Second, and this is the operative conclusion, the goal of unit testing cannot be achieved by just adding as many tests as possible: you need to consider both the value of a test and its maintenance cost.

8. Code coverage and its criteria

Code coverage is a measure describing the degree to which the source code of a program has been tested. It is a form of white box testing: it finds the areas of the program not covered by a set of test cases, it forces the creation of some test cases to increase coverage, it offers a quantitative measure, and it helps to measure the efficiency of test implementation. It contrasts with black box testing, where of a component we only care about its external behaviour.

The criteria listed are function coverage, statement coverage, branch coverage and condition coverage, among others. The deck works them on one tiny function, and the numbers it gives are the ones to remember.

int f(int a, int b) {
  if (a > 0 && b > 0) {
     int res = a + b;
     return res;
  }
  return -1;
}

The deck then shows what a full-condition-coverage test looks like when written out, on an Adder class with the same shape:

package coverage;

public class Adder {
    public int add(int i1, int i2) {
        if (i1 > 0 && i2 > 0) {
             return i1 + i2;
        }
        return -1;
    }
}

package coverage;

import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

class AdderTest {

    @Test
    void testAddition() {
        var adder = new Adder();
        assertEquals(30, adder.add(10, 20));
        assertEquals(-1, adder.add(-1, 20)); // test needed for full coverage
        assertEquals(-1, adder.add(20, -1)); // test needed for full coverage
    }
}

9. Parameter value coverage

Coverage of code is not coverage of inputs, and parameter value coverage (PVC) is the criterion that says so. It requires that, in a function with parameters, all the common values for those parameters have to be considered: given the type or class of a parameter, a set of possible testing values is to be tested.

The example given is void f(String s){...}, which should be tested assuming s equals: null, empty, whitespace (space, tab, newline), a valid string, an invalid string, a single-byte string, a very long string, and so on. Failure to test each possible parameter value may leave a bug. And here is the punchline: testing only one of these could result in 100 percent code coverage, since each line is covered, but as only one of seven options is tested, there is only 14.2 percent PVC.

10. Test doubles: the reference model

Unit tests target single components, working in isolation. Otherwise we would be testing a collaboration, and failures would be about the collaboration. Incrementality and modularity necessarily call for testing components separately first, and then their collaboration.

So how do we unit-test a component that needs others to function correctly? We rely on a test double: an alternative implementation for the component we depend upon, quickly set up just for the sake of the tests and with predictable behaviour. Two rules follow immediately. Test doubles must be free of errors by construction, otherwise we fall back into integration-testing problems; and if testing is really needed for them, it should be much, much simpler than testing the actual version.

The reference model uses three acronyms consistently, and the exam uses them too:

• SUT: system under test.
• DOC: depended-on component.
• TD: test double, an alternative version of the DOC.

The testing framework interacts with the SUT, which calls the DOC or the test double, and hence verifies what happened.

Why bother

Sometimes it is just plain hard to test the SUT because it depends on other components that cannot be used in the test environment: they are not available, they will not return the results needed for the test, executing them would have undesirable side effects, they would be too slow or costly, or they would break automation. In other cases, the test strategy requires more control or visibility of the internal behaviour of the SUT: is the SUT calling the DOC in the proper way? Is it correctly dealing with an error reported by the DOC? What would happen if the DOC replied with an exception?

The metaphor offered is the stunt double of the movie industry: stunt doubles may not be able to act, but they know how to fall from great heights or crash a car. During the fixture setup phase we replace the real DOC with our TD; depending on the kind of test we may hard-code the behaviour of the TD, or configure it during setup. When the SUT interacts with the TD, it will not be aware that it is not talking to the real DOC.

Two problems doubles solve, and one they cause

Doubles help with an Untested Requirement, which we cannot verify because neither the SUT nor its DOCs provide an observation point for the indirect output we need to verify. The deck's example is a FlightManager whose requirement is that each action gets correctly logged:

@Test
public void testRemoveFlight() {
  var fm = new FlightManager();
  var flight = fm.createARegisteredFlight();
  fm.removeFlight(flight.getFlightNumber());
  assertFalse(fm.flightExists(flight.getFlightNumber()));
}

class FlightManager {
  ...
  public void removeFlight(long flNumber) {
    dataAccess.removeFlight(flightNumber);
    log("CreatedFlight", flightNumber); // Bug!
  }
}

The bug in the log message cannot be caught, since there is no testability for that requirement: the SUT has behaviour not visible through its public interface, and expected side effects that cannot be directly observed by the test. The solution is to test against a test double for the Logger, tracking log actions, which implies the need for a clearly separated logger.

They also help with Untested Code, where a DOC does not provide the control point to exercise the SUT with the necessary indirect inputs:

public String getCurrentTime() {
  try {
    return timeProvider().getTime();
  } catch (Exception e) {
    return e.Message;
  }
}

How do we test that when the timeProvider throws, the SUT handles it correctly? The SUT has code paths reacting to particular DOC behaviour that we have found no way to exercise; typically the DOC is called in ways that hardly result in complete covering of the SUT, often because the SUT interface legitimately does not capture all internal events. The solution is a test double for timeProvider that always raises the exception.

The applicability, meanwhile, is pervasive. Components typically replaced with a double include GUIs and console UIs, random generation, complex algorithms, external resources (databases, files, network, web resources), untested code such as legacy, bad or early code and interfaces with no implementation yet, and in fact any dependency at all.

11. The taxonomy of doubles

Five variations, grouped in two categories. The grouping is the important part, and the deck states the criterion crisply.

And in detail:

• Dummy: replaces the DOC with an untested reference, an "empty object".
• Stub: replaces the DOC with a predefined responder, method per method.
• Fake object: replaces the DOC with a light-weight version of it, giving a sequence of results.
• Spy: a proxy for a DOC, used to track the calls it receives.
• Mock: replaces the DOC with an object used to check whether it is overall correctly used by the SUT.

Creating test doubles by hand is typically time consuming, which often hampers applicability, so good libraries to create them are key. The one used here is Mockito.

FailingPolicy dummyFailingPolicy = mock(FailingPolicy.class);
device = new StandardDevice(dummyFailingPolicy);
// checking that a device is on will not affect the strategy
assertFalse(device.isOn());

when(this.stubFailingPolicy.attemptOn()).thenReturn(true);
device.on();
assertTrue(device.isOn());

when(this.fakeFailingPolicy.attemptOn()).thenReturn(true, true, false);
when(this.fakeFailingPolicy.policyName()).thenReturn("mock");

this.spyFailingPolicy = spy(new RandomFailing());
device = new StandardDevice(this.spyFailingPolicy);
verifyNoInteractions(this.spyFailingPolicy);

verify(this.mockFailingPolicy, times(0)).attemptOn();
device.on();
verify(this.mockFailingPolicy, times(1)).attemptOn();

12. Mockito at work

Mockito is just a library to create stubs and mocks, hosted at github.com/mockito/mockito, added with the SBT dependency "org.mockito" % "mockito-core" % "3.+" % Test. Its key basic constructs are four:

• mock(C.class) to create a dummy implementation;
• when(mockObj.meth()).thenResult(res) to instruct a stubbed reply;
• spy(obj) to insert a wrapper that detects events;
• verify(spyObj).meth() to check whether meth has been called on spyObj.

The example that carries the section extends the Device of section 5 with a collaborator, a failing policy, so that there is finally something to double.

public interface Device {
    void on() throws IllegalStateException;
    void off();
    boolean isOn();
    void reset();
}

public interface FailingPolicy {
    boolean attemptOn();
    void reset();
    String policyName();
}

import java.util.Objects;

public class StandardDevice implements Device {
    private FailingPolicy failingPolicy;
    private boolean on = false;

    public StandardDevice(FailingPolicy failingPolicy) {
        this.failingPolicy = Objects.requireNonNull(failingPolicy);
    }
    // ...
}

The test class is organised with @Nested showcases, one per kind of double, which is itself an illustration of the organisation guidelines from section 6.

The fake showcase deserves a second look, because it is where a double stops being a constant and starts being a small state machine:

@Nested
class ShowcaseFakes {
    private FailingPolicy fakeFailingPolicy;

    @BeforeEach
    void init() {
        this.fakeFailingPolicy = mock(FailingPolicy.class);
        device = new StandardDevice(this.fakeFailingPolicy);
        // faking is more than stubbing: this object pretends to be the real one
        when(this.fakeFailingPolicy.attemptOn()).thenReturn(true, true, false);
        when(this.fakeFailingPolicy.policyName()).thenReturn("mock");
    }

    @Test
    @DisplayName("Device switch on and off until failing")
    void testSwitchesOnAndOff() {
        IntStream.range(0, 2).forEach(i -> {
               device.on();
               assertTrue(device.isOn());
               device.off();
               assertFalse(device.isOn());
        });
        assertThrows(IllegalStateException.class, () -> device.on());
    }
}

And the spy showcase shows the other half of Mockito, the part that inspects rather than instructs, including the powerful mockingDetails API:

@Nested
class ShowcaseSpies {
    private FailingPolicy spyFailingPolicy;

    @BeforeEach
    void init() {
        // the spy is essentially a proxy to the DOC, used to capture events
        this.spyFailingPolicy = spy(new RandomFailing());
        device = new StandardDevice(this.spyFailingPolicy);
    }

    @Test
    @DisplayName("AttemptOn is called as expected")
    void testReset() {
        device.isOn();
        // no interactions with the spy yet
        verifyNoInteractions(this.spyFailingPolicy);
        try {
                device.on();
        } catch (IllegalStateException e) {}
        // has attemptOn been called?
        verify(this.spyFailingPolicy).attemptOn();
        device.reset();
        // have at least two method invocations been made?
        assertEquals(2,
                Mockito.mockingDetails(this.spyFailingPolicy).getInvocations().size());
    }
}

Finally, the same doubles can be declared with annotations rather than factory calls, which keeps the fixture shorter:

public class AlternateStandardDeviceTest {

    private Device device;
    @Mock FailingPolicy stubFailingPolicy;
    @Spy RandomFailing spyRandomPolicy;

    @BeforeEach
    void init() {
        MockitoAnnotations.openMocks(this);
    }

    @Test
    void testMock() {
        device = new StandardDevice(this.stubFailingPolicy);
        // multiple stubbing
        when(this.stubFailingPolicy.attemptOn()).thenReturn(false);
        when(this.stubFailingPolicy.policyName()).thenReturn("mock");
        assertThrows(IllegalStateException.class, () -> device.on());
        assertEquals("StandardDevice{policy=mock, on=false}", device.toString());
    }

    @Test
    void testSpy() {
        device = new StandardDevice(this.spyRandomPolicy);
        // no interactions with the spy yet
        verifyNoInteractions(this.spyRandomPolicy);
        try {
                device.on();
        } catch (IllegalStateException e) {}
        // has attemptOn been called?
        verify(this.spyRandomPolicy).attemptOn();
    }
}

13. Integration testing

Unit testing is not enough. We can never be sure a system works as a whole if we rely on unit tests exclusively: they are great at verifying business logic and at guiding a quality development process, but not enough to test higher-level behaviour. We have to validate how different parts of a system integrate with each other and with external systems, and this has to be carried on in an automated way even if it seems unfeasible because of the unavailability of external resources. Crucially, integration tests are not system tests: we just have to isolate collaborations. And it is crucial to balance the number of unit and integration tests.

The worked example

The system implements one feature: changing the user email. It retrieves the user and the company from the database, delegates the decision-making to the domain model, and then saves the results back to the database and puts a message on the bus.

public class UserController {
  private final Database database;
  private final MessageBus messageBus;

  public UserController(Database database, MessageBus messageBus) {
    this.database = database;
    this.messageBus = messageBus;
  }

  public void changeEmail(int userId, String newEmail) {
    User user = UserFactory.create(database.getUserById(userId));
    if (!user.canChangeEmail()) {
      throw new IllegalStateException("user can't change email");
    }
    Company company = CompanyFactory.create(database.getCompany());
    user.changeEmail(newEmail, company);
    database.saveCompany(company);
    database.saveUser(user);
    user.emailChangedEvents.forEach(ev ->
      messageBus.sendEmailChangedMessage(ev.userId(), ev.newEmail()));
  }
  ...
}

@Test
public void testChangingEmail() {
  // Arrange
  var db = new Database(connectionString);
  User user = createUser("[email protected]", UserType.EMPLOYEE, db);
  createCompany("mycorp.com", 1, db);
  var messageBusMock = mock(MessageBus.class);
  var userController = new UserController(db, messageBusMock);

  // Act
  userController.changeEmail(user.userId(), "[email protected]");

  // Assert DB state
  Object[] userData = db.getUserById(user.userId());
  User userFromDb = UserFactory.create(userData);
  assertEquals("[email protected]", userFromDb.email());
  assertEquals(UserType.CUSTOMER, userFromDb.type());

  // Assert Company state
  Object[] companyData = db.getCompany();
  Company companyFromDb = CompanyFactory.create(companyData);
  assertEquals(0, companyFromDb.numberOfEmployees());

  // Check interaction with the mock
  verify(messageBusMock, times(1))
    .sendEmailChangedMessage(user.UserId(), "[email protected]");
}

The point the deck insists on is the one drawn on the right of the plate: it is important to check the state of the database independently of the data used as input parameters. The test queries the user and company data separately in the assert section, creates new userFromDb and companyFromDb instances, and only then asserts their state. This approach ensures that the test exercises both writes to and reads from the database, and thus provides the maximum protection against regressions.

Best practices

Four general guidelines help get the most out of integration tests: making domain model boundaries explicit; reducing the number of layers; eliminating circular dependencies; and using multiple act sections in a test. As usual, practices that are beneficial for tests also tend to improve the health of the code base in general, and specifically writing integration tests earlier supports so-called design for testability.

14. Quality, clean tests and the mnemonics

The module closes by placing testing inside the wider quality picture, cherry-picking, as the deck says, some key aspects:

• Code defects: needless repetitions and complexity, rigidity, fragility, immobility, viscosity, opacity.
• Software engineering principles: rigour and formality, separation of concerns, modularity, abstraction, anticipation of change, generality, incrementality.
• Principles of good OOP programming: KISS and DRY, plus SOLID (SRP, OCP, LSP, ISP, DIP).
• Design patterns: template recipes to inject good principles in a design.
• Guidelines: code conventions, rules of effective programming, rules of clean coding.
• Methodology: agility, test-driven and agile development, continuous integration and advanced tools.
• Advanced paradigms: modern directions in programming style, for example functional idioms in OOP.

Clean tests

The distilled rules are four. Three things to pay attention to: readability, readability, readability. Simplify tests by creating a higher-level API, in the form of a DSL or facade. One assert per test, but only if that does not violate KISS. One concept per test, namely a single reason to fail, which is the single responsibility principle applied to tests.

Right-BICEP, and checking correctness

Two further mnemonics complete the toolkit. Right-BICEP asks: are the Right results right, identifying significant correct paths; Boundary conditions, is it working correctly in edge cases such as methods not called, negative numbers, null references; Inverse relationships, apply the inverse behaviour to the result and see if you get the input back; Cross-checks by other means, using as expected result something obtained by other algorithms, implementations or techniques; Error conditions, force the system into error and check things are properly handled; Performance bounds, compute how long some code takes and fail or stop if too long, and generally check performance, memory and network if you have requirements on them.

The CORRECT checks then enumerate what to check about a value: Conformance, is an input value correctly formatted; Ordering, are values provided in the right order and are methods called in the proper order; Range, are numeric values in the proper range; Reference, are we referencing external components; Existence, check null references, empty strings, empty collections; Cardinality, is the number of elements in a collection right; Time, check timeouts and concurrency aspects.

A methodology for TDD, and testable code

The suggested methodology is: analyse requirements and turn them into a list of tests, one phrase each; turn the list of tests into a fixture and method names for test cases; select each time the next test to implement and make pass, using the criteria start with the happy path, simple first, most progress with little effort. Two overall strategies are possible: a breadth-first approach, starting with a global-level interface, faking all its tests, then proceeding to the lower layer; or a depth-first approach, starting with a global-level interface and implementing one test immediately, proceeding on the lower layer. Conclude with the corner cases, that is, with the CORRECT checks.

Finally, the suggestions to write testable code, which read as a compact design guide: public APIs are contracts, do not change them; reduce dependencies; create simple constructors and initialise in methods; follow the principle of least knowledge; avoid hidden dependencies and global state; generalise and generify APIs as much as possible; favour composition over inheritance; favour polymorphism over switches.

15. Lab: mocking, isolation and integration

The lab of this module is deliberately small in scope and large in consequence: it asks you to take an existing, badly separated test suite and split it into genuinely isolated unit tests plus explicit integration tests.

References

• Mockito: https://site.mockito.org/
• Lab repository 02-repo-testing: https://github.com/mviroli/asmd-public-02-testing
• LLMs to experiment with: ChatGPT, Perplexity, Claude, Gemini
• Slide deck 02 provides the JUnit and Mockito examples reproduced above

The general goals are to be operative with Mockito and JUnit, to understand mocking in its details, to exercise integration testing, and to pre-check the ability of an LLM to help you in unit and integration testing.

Operational steps

Step 1, get ready: clone the repository, open it in IntelliJ, run the tests; if everything is fine you are ready to go.

Step 2, reorganise the Device example, with the goal of creating technically excellent, isolated tests for the device and its failing policy. First explore the existing code: review test/java/devices/StandardDeviceTest.java and test/java/devices/AlternateStandardDeviceTest.java, identify the different mocking techniques and patterns used in these files, and understand how Mockito is being applied in the different scenarios. Then create proper isolated unit tests: develop a new unit test class for the device, completely isolated from the failing policy, and a separate unit test class focused solely on testing the failing policy, ensuring both follow best practices. Finally build integration tests that verify the proper collaboration between the device and the failing policy, and compare the differences between your unit and integration tests.

R&D tasks

Test your knowledge