The context is the one the whole course has built: robot's behaviour emerges from the interaction of three components — the control software, the physical structure of the robot, and the environment. Even in the extreme case of static environments and deterministic conditions, robot behaviour cannot be fully predicted (e.g. the initial condition might impact the final result). Therefore, the evaluation of a control software must be statistically assessed — as for any experiment in science.
Chapter 1 promised: "behaviour must be measured statistically, not asserted" (the sensory-motor loop makes prediction from the controller alone impossible). Chapter 4 added: since no controller is universally best, a design claim cannot be settled by argument — it must be settled empirically. This chapter delivers the methodology: state the hypothesis, sample the environment distribution, measure a stated merit factor, run independent replicas, apply a statistical test. Everything else in the course — evolutionary robotics, automatic design, robot learning — is validated with exactly this protocol.
The scientific method deck frames it with typical questions: is the robot's behaviour predictable? Can it be modelled? Is it stable? Is this behaviour different from that one, or is there no significant difference between them? Which program performs better (where "better" is defined by some measurable criterion)? A theory is a coherent body of generalisations and principles forming the frame of reference within which research is conducted; its two key elements: it allows the formulation of hypotheses for testing, and it makes predictions, serving as a safeguard against unfounded or weakly supported assumptions.
The observation attached to the list: the experimental evaluation of a robot's control software is analogous to the validation phase in a machine learning context — unless both initial conditions and environment are always exactly the same, the validation phase is carried out considering a different set with respect to that of the training phase. (Chapter 11's "training and test sets" is the same principle, now stated as a general rule.)
The deck's running example: task — phototaxis with collision avoidance; environment — static, but unknown, and noisy; we only know in advance type and number of obstacles, but not their position; robot initial placement and orientation — unknown; F — distance from the light after T seconds.
The choice of the merit factor should quantitatively capture the criteria we think are important for evaluating the behaviour. In this case, we assume that T seconds is the time we allow in our real-world application. Note the shape of the argument: F is not "whatever is easy to measure" — it is a contract between the experiment and the application.
The practical approach:
In the example: five control software instances P1..P5, 100 independent replicas for each → 100 evaluations each. A random setting is generated for each run; this inevitably introduces variance because the different control programs are evaluated across different conditions — but if the number of runs is sufficiently high, the procedure returns robust results.
The simple comparison: suppose the average distances are 0.79, 0.76, 0.26, 0.42, 2.92 for P1..P5. Just by looking at the averages, can we assert with certainty that P3 is the best? And that P5 is the worst? Without analysing the data and making statistical tests, we cannot.
In any experimental science, we perform experiments designed to test our hypothesis; the experiment is run, data describing the relevant aspects are logged and subsequently analysed. One typically wants to answer: is there a statistically significant correlation between input and output variables? Is there a statistically significant difference between the experimental results and some baseline (either another method, or a well-established mechanism that is well understood)? Alternatively: could the experimental results be explained by random events? Are they a fluke?
The steps of experimental analysis: (1) define the objectives and formulate the hypotheses; (2) define the experimental setting; (3) analyse the results (with statistics and plots); (4) statistically assess the hypotheses.
On hypotheses: a hypothesis is a formally stated expectation about a behaviour that defines the purpose and the goals of a study; it defines, explains, and guides the research. Without a clear hypothesis in the beginning it is impossible to do good research. Every time we design control software for a given task, we implicitly formulate a hypothesis; a principled approach requires that the hypothesis is posted explicitly. Examples: "given the actual arena, the robot starting in a random position will follow walls keeping them on the right"; "controller A performs better than controller B w.r.t. utility function F".
Observations without theory are impossible, as observations are always the consequence of a hypothesis or a question. The aim of scientific investigation is to refute a hypothesis, and all experimentation is geared towards that goal. If a hypothesis withstands all attempts of refutation, it is tentatively adopted as true, but not considered proven. The only truth that can be firmly established is that a theory is false, never that it is true. This is why the statistical test of section 7 is framed as rejecting the null hypothesis — not as proving the alternative.
And the deck's "robotic science" point: without theoretical understanding, any design process is largely dependent upon trial-and-error experimentation and iterative refinement. A hypothesis — a justified expectation — guides the design process; results obtained are fed back into the process and brought into alignment with the theory. Every process of designing technical artefacts is based on some kind of assumptions (a theory), even if very little is known about the object being designed.
| Trap | What it is | Example from the deck |
|---|---|---|
| Confounding effects | The phenomenon of interest is closely correlated with some other effect of no interest; the experiment must be designed so that only the factor of interest is investigated | Measuring whether obstacle avoidance movements become smoother with time, without realising the improvement could be the decreasing battery charge making the motors sluggish |
| Floor and ceiling effects | The experimental design is too demanding or too simple to highlight relevant phenomena | Two service robots compared in too simple environments show no difference (floor effect); in too complicated environments neither performs satisfactorily (ceiling effect); just the right environment complexity is needed |
| Systematic errors | Constant errors (biases) can mask true effects | Asymmetries between actuators make a differential-drive robot always swerve to one side when allegedly moving straight |
| The conspiracy of wishful thinking | Losing objectivity: researching selectively to attain a desired outcome, or assuming our expectations are correct without testing them | Drawing two wrong conclusions from an untested assumption: (a) if anything goes wrong the cause is elsewhere; (b) the behaviour could be even worse without that magic component (wrong counterfactuals) |
Countermeasures: properly and precisely define the experiments; randomisation; statistical methods to assess the performance and other properties of the robot's behaviour.
Four short experimental scenarios from the deck's examples. Pick the trap each one illustrates; the widget confirms or corrects you.
The distribution of the results is inspected with histograms. Then the deck moves to robust summaries. Median (from Cohen, Empirical methods for AI): "If the values of the sample are sorted into a nondecreasing order, the median is the value that splits the distribution in half. If the number of values is even, the sample has two middle values, and the median can be found by interpolating between them or by selecting one of them arbitrarily." In the example: median(P1) = 0.118, median(P2) = 0.118, median(P3) = 0.023, median(P4) = 0.023, median(P5) = 2.994.
Quartiles: quantiles that partition the values into four (approximately equal) parts. The 1st quartile is the value x such that 25% of values are ≤ x; the 2nd is the median; the 3rd is the value x such that 75% of values are ≤ x. Boxplots provide a graphical view of the important parameters of the distributions: central segment = median; edges = 1st and 3rd quartiles; whiskers = computed as a function of the quartiles, marking the limits of outlier values; circles = outliers.
In empirical computer science, an outlier is usually not produced by noise, a mistake in running a replica, or a fluctuation in the experimental equipment. Therefore it should attract our attention instead of being discarded. A robot that occasionally crosses the whole arena in a phototaxis experiment is not a measurement error — it is a behaviour, and possibly an interesting one.
Null hypothesis: the hypothesis that the observed outcome of an experiment is due to chance alone, and not due to a systematic cause. Statistically significant is a precisely defined technical expression: the outcome of an experiment differs from the null hypothesis by more than what could be attributed to random fluctuations.
Statistical tests involve: (1) determining what kind of data are to be analysed; (2) determining what kind of questions are being asked; (3) selecting an appropriate test; (4) performing the analysis. Several kinds exist, depending on the type and properties of the data collected:
Parametric tests can be applied to data with a known distribution (typical case: Normal). Non-parametric tests can be applied to any set of data — more robust but less powerful than parametric tests. Usually, results from robots' dynamics are not normally distributed, or samples are so few that a parametric test cannot be applied, so a frequent choice is the Wilcoxon test. Its null hypothesis: the median of the two distributions is the same; the alternative hypothesis: it is not. "This is usually what we wish — maybe because we have designed a new controller that we expect to work better than the others — but we must avoid projecting our wishes into the interpretation of the data, and we should ask the same integrity to scientists presenting their own results."
The most important value returned by the test is the p-value: a measure of how likely the sample results are, assuming the null hypothesis is true. The smaller the p-value, the less likely the sample results under the null hypothesis. The sound approach: set a parameter α; if p-value < α, the null hypothesis can be rejected; otherwise, it cannot be rejected — typically α = 0.05 or 0.01. Two remarks worth memorising:
Choose α = 0.01. wilcox.test(res_P1, res_P2) returns p-value = 0.661: we cannot reject the null hypothesis — not sufficient data to claim that P1 and P2 produce different behaviours. Applying the test to each pair yields a triangular matrix of results:
| P2 | P3 | P4 | P5 | |
|---|---|---|---|---|
| P1 | 0.661 | < 10−15 | < 10−15 | < 10−15 |
| P2 | — | < 10−15 | < 10−15 | < 10−15 |
| P3 | — | — | 0.04 | < 10−15 |
| P4 | — | — | — | < 10−15 |
(p-values < α are highlighted in red in the slides.) For the cases with p-value < α we can reject the null hypothesis. The resulting ranking: {P3, P4} ≺ {P1, P2} ≺ P5.
The final ranking refers to the choice of the statistical parameter (median), the statistical test (Wilcoxon) and α (0.01). P5 seems to be the worst — and it must be, as it is just a random walk! The relation between P1 and P2 is quite clear: they are the same subsumption controller implemented with different behaviour trees (the equivalence Chapter 8 proved structurally, here confirmed statistically: p = 0.661). The relation between P3 and P4 is less clear (two versions of a motor schema controller, the former equipped with a tangential field), but there are no elements suggesting they differ significantly.
Synthetic samples reproducing the deck's medians (0.118, 0.118, 0.023, 0.023, 2.994). Each run of the test recomputes the p-values with the Mann-Whitney normal approximation; watch which pairs stay non-significant and which are decisive.
Parameter tuning is typical of most fields of AI, and it is often still addressed via manual trial and error (mostly ineffective!). Automatic techniques exist and are undoubtedly effective; a principled and theoretically grounded approach is provided by Birattari (Tuning Metaheuristics: A machine learning perspective, Springer, 2009).
F-Race (Chapter 11, section 8) is a method for offline configuration of parameterised algorithms: in the training phase, a parameter configuration is determined in a limited amount of time that optimises some measure of algorithm performance; the final configuration is then deployed in a production phase where the algorithm solves previously unseen instances. It is a problem of generalisation, as in machine learning: based on a training set of instances, find configurations that perform well on a potentially infinite set of unseen instances. The main idea — run the configurations iteratively on new sampled instances and progressively discard the statistically significantly worse ones — avoids wasting computational time on bad configurations. R code: https://cran.r-project.org/package=irace.
CMA-ES (Covariance Matrix Adaptation Evolution Strategy): the parameters are considered as variables of an optimisation problem whose objective function is the performance of the resulting control program; the variables are iteratively sampled according to a distribution that is adapted on the basis of the most promising previous samples.
The F-Race section here is the same engine as AutoMoDe's: the configuration problem is solved by racing, and the "tuning" of section 9 is exactly the "design" of Chapter 11 at a different scale — tune parameters of a fixed architecture vs design the architecture itself. The methodological continuity is the point: automatic design is parameter tuning with the architecture in the search space.
Besides controlled experiments, in autonomous robotics there is also room for explorative experiments (see the works by Viola Schiaffonati, with Francesco Amigoni). An explorative experiment is intended as a form of investigation carried out in the absence of a complete theory or theoretical background, where the control of the experimental factors cannot be fully managed from the beginning. For example: we want to observe the behaviour of a robot controlled by the same software across different kinds of environments.
Amigoni and Schiaffonati motivate the notion from the current practice: in a significant sample of award-winning papers at ICRA and IROS, few experiments come close to controlled experiments in the traditional sense. The term "experiment" in computing covers at least five different views: feasibility experiments (demonstrating that a technology works), trial experiments (evaluating aspects of a system with predetermined variables in a laboratory), field experiments (outside the laboratory, in the real world), comparison experiments (comparing solutions to find the best for a problem), and controlled experiments (aimed at generalisation and prediction). Explorative experiments give reasons of a part of the current experimental activity that cannot be satisfactorily accommodated under the traditional concept of controlled experiment.
Do not read this section as a licence to skip statistics. The chapter's message is two-sided: when a controlled comparison is possible, it is compulsory (the Wilcoxon protocol); and when the theory is not yet there — exploration comes first, and it must be reported as exploration, not dressed up as a controlled experiment. Explorative experiments are how hypotheses are born; controlled experiments are how they are tested. The dynamical-systems tradition (the deck names it as a viable theoretical background for autonomous robotics, alongside complex systems and information theory) is what the course's own language — attractors, bifurcations, feedback — has been using all along.
Because robot behaviour emerges from the interaction of control software, physical structure and environment, and even in static, deterministic conditions it cannot be fully predicted (e.g. the initial condition may impact the final result). The evaluation of a control software must therefore be statistically assessed, as for any experiment in science.
Some control software instances P1..Pn; an evaluation function F implementing a merit factor (maximised or minimised); a description of the environment (known in advance? what characteristics are known?); a description of the initial condition of the robot (same position and orientation?).
Task: phototaxis with collision avoidance; environment static but unknown and noisy (type and number of obstacles known, positions not); initial placement unknown; F = distance from the light after T seconds. Protocol: devise a model for instantiating the objects to generate samples from a distribution; run m independent experiments collecting the value of F; state in advance the scientific hypothesis and the final goal (best on average? minimal guaranteed quality? best achievable?). The example: five controllers, 100 replicas each.
Because the random settings introduce variance and the difference between averages may be due to chance. Without analysing the data and making statistical tests, we cannot assert which controller is best or worst — the significance of a difference must be assessed with a statistical test.
Steps: (1) define the objectives and formulate the hypotheses; (2) define the experimental setting; (3) analyse the results with statistics and plots; (4) statistically assess the hypotheses. A hypothesis is a formally stated expectation about a behaviour that defines the purpose and goals of a study; it must be posted explicitly, and every time we design control software we implicitly formulate one.
Observations are always the consequence of a hypothesis or a question. The aim of scientific investigation is to refute a hypothesis; if a hypothesis withstands all attempts of refutation it is tentatively adopted as true, but never proven. The only truth firmly established is that a theory is false, never that it is true — hence tests are framed as rejecting the null hypothesis, not proving the alternative.
Confounding effects (correlated uninteresting factor — e.g. battery charge instead of learning); floor and ceiling effects (environment too simple or too demanding to reveal differences); systematic errors (constant biases — e.g. actuator asymmetry making the robot swerve); the conspiracy of wishful thinking (researching selectively toward a desired outcome, untested assumptions, wrong counterfactuals). Countermeasures: properly and precisely define the experiments, randomisation, statistical methods.
Median: the value that splits the sorted sample in half (interpolating or picking one middle value when even). Quartiles: quantiles partitioning the distribution into four parts — 1st: 25% of values ≤ x; 2nd: the median; 3rd: 75% ≤ x. Boxplot: central segment = median, edges = 1st and 3rd quartiles, whiskers computed as a function of the quartiles marking the limits of outlier values, circles = outliers. In empirical computer science an outlier should attract attention, not be discarded.
Null hypothesis: the observed outcome is due to chance alone, not to a systematic cause. Statistically significant: the outcome differs from the null hypothesis by more than what could be attributed to random fluctuations. Parametric tests (e.g. t-test) apply when the distribution is known (typically Normal); non-parametric tests (e.g. Wilcoxon/Mann-Whitney) apply to any data — more robust but less powerful. Robot dynamics results are usually not normally distributed, so non-parametric tests are frequent.
The Wilcoxon (Mann-Whitney) test's null hypothesis is that the medians of the two distributions are equal; the alternative that they are not. The p-value measures how likely the sample results are assuming the null hypothesis; if p < α (typically 0.05 or 0.01) the null hypothesis can be rejected, otherwise it cannot. Two caveats: not rejecting does not imply equality — just no evidence of difference; and the p-value does not estimate the strength of the difference, only its significance.
With α = 0.01: P1–P2 p = 0.661 (cannot reject: the same subsumption controller with different BTs — statistically indistinguishable); P3–P4 p = 0.04 (cannot reject at α = 0.01: two versions of a motor schema controller, no significant difference); all other pairs p < 10⁻¹⁵ (reject: significant differences). Ranking: {P3, P4} ≺ {P1, P2} ≺ P5 — P5 is just a random walk. The ranking refers to the choice of median, Wilcoxon test and α.
Parameter tuning is typical of AI and often done by ineffective manual trial and error; automatic techniques exist and are effective. F-Race is an offline configuration method: run candidate configurations iteratively on new sampled instances and discard the statistically significantly worse ones (generalisation as in machine learning; R implementation: irace). CMA-ES treats parameters as variables of an optimisation problem whose objective is the performance of the control program, sampling them according to a distribution adapted on the basis of the most promising previous samples. Automatic design is parameter tuning with the architecture in the search space.
Explorative experiments are a form of investigation carried out in the absence of a complete theory or theoretical background, where the control of the experimental factors cannot be fully managed from the beginning (e.g. observing a robot controlled by the same software across different kinds of environments). They account for a significant part of current practice that does not fit the traditional controlled-experiment notion (whose family includes feasibility, trial, field, comparison and controlled experiments). Exploration is how hypotheses are born; controlled experiments are how they are tested.