Meta-models, DSLs and code generation

1. Meta-modelling nomenclature

The lecture goals are stated in three lines: understand metamodelling, understand domain specific languages, practice with model driven development in Xtext. The first goal needs a vocabulary that is built by asking the same question three times.

Three remarks accompany the definition of model, and they explain why the word is used differently here than in ordinary speech: the model abstracts a number of similar systems rooted in the domain; each system is an instance of the model it has been designed from; a model is a template for several systems.

The ladder is generated by one recurring question — in which language is this expressed? — applied first to the model, then to the meta-model. It stops at MOF by convention, because MOF is expressed in terms of itself.

2. Why meta-models are important

The slide devoted to this question is short and unusually practical:

• Meta-models are the very first thing you should try to identify whenever approaching a new technology.
• If you grasp the meta-model, you grasp the essence of the technology — which may be the same for many other technologies.
• For example, after learning the basics of OOP (classes, methods, objects) you may easily learn any other OOP language, by simply asking yourself how each meta-model element is expressed in the new language: how are classes, methods and objects expressed here?
• When you model a domain (for instance with DDD) you are always exploiting some meta-model, whether you are aware of it or not.

3. Model-driven whatever, and code generation

The lecture acknowledges the terminological fog directly: several slightly similar names create confusion — model-driven engineering / development / architecture — and it recommends reading Martin Fowler's article on Model-Driven Software Architecture to clarify. Despite the names, the key ideas are two:

• The software engineering workflow should start by modelling the domain at hand carefully (for instance with DDD), as opposed to focussing on algorithms and data structures.
• The production of a runnable implementation should be automated as much as possible (for instance by generating code from models), as opposed to writing code by hand.

The second idea has a prerequisite, spelled out on the following slide. The assumption is that the model is expressed by means of some formal language, where formal ≈ interpretable by a machine, and:

Formality is a prerequisite for automation.

The chain that formality enables has three links: the model is parsed by a machine; the model is transformed into another formal language (for instance an OO programming language); the transformed model is rendered into a file (for instance source code).

Is UML adequate? Is it the only choice?

The lecture answers with a qualified no:

• UML has a formal syntax and semantics (reified into graphical representation rules), but these are rarely enforced by software tools.
• Furthermore, UML focuses on software, and is therefore only practical for software engineers.

The second objection is the fatal one for DDD purposes: a notation that domain experts cannot read cannot be the place where the ubiquitous language lives. That is the opening through which domain-specific languages enter.

4. Domain-specific languages

Domain-specific languages (DSL) are programming, description or specification languages targeting one particular class of problems: they are not meant to address all possible problems, but just the ones they are designed for. This is the opposite of general-purpose languages (GPL), which target as many classes of problems as possible — the programming languages you have learned so far are GPLs.

And then the sentence that ties this chapter back to Chapter 2: DSLs may act as custom meta-models for a given domain. Where UML gives you one fixed meta-model designed for software engineers, a DSL lets you define a meta-model whose vocabulary is the ubiquitous language of your domain.

Examples of DSLs you may already know: regular expressions for text processing, SQL for database querying, CSS for styling web pages, HTML for describing web page content, DOT for graph visualisation, PlantUML for UML diagram visualisation, Gherkin for Behaviour-Driven Development, VHDL for hardware description.

digraph G {
    size ="4,4";
    main [shape=box];
    main -> parse [weight=8];
    parse -> execute;
    main -> init [style=dotted];
    main -> cleanup;
    execute -> { make_string; printf}
    init -> make_string;
    edge [color=red];
    main -> printf [style=bold,label="100 times"];
    make_string [label="make a string"];
    node [shape=box,style=filled,color=".7 .3 1.0"];
    execute -> compare;
}

Scenario: Verify withdraw at the ATM works correctly
Given John has 500$ on his account
When John ask to withdraw 200$
And John inserts the correct PIN
Then 200$ are dispensed by the ATM
And John has 300$ on his account

DFF : process(RST, CLK) is
begin
if RST = '1' then
Q <= '0';
elsif rising_edge(CLK) then
Q <= D;
end if;
end process DFF;

pension contribution InpsTerziario paid by owner {
    considered_salary = (taxable of IRES for employer - amount of IRES for employer
                         - amount of IRAP for employer) by ownership share
    rate = brackets [to 46,123] -> 22.74%,
                    [to 76,872] -> 23.74%,
                    [above]     -> 0%
    amount = (rate for considered_salary) with minimum 3,535.61
}

Benefits of adopting a DSL

• To communicate with domain experts in their own language — Gherkin, for example, can be understood by both engineers and stakeholders.
• To let domain experts write the specifications (that is, the model) of the system they want, hence reducing ambiguities among stakeholders and engineers.
• To focus on the domain rather than on the implementation.
• To hide the implementation details from the domain experts, exposing only business-related concepts at the domain level.

5. DSL versus GPL

DSLs are not a replacement for GPLs: they are complementary. The difference is admittedly fuzzy, so the lecture clarifies it with a comparison along eight dimensions.

6. DSL engineering: semantics and execution

Most often the focus is on the syntax of a DSL, because that is how users perceive it. Yet the semantics is equally important: that is what dictates how the DSL works, and it is what engineers (DSL implementers) focus upon. Intuitively, semantics is given to languages by writing the machinery supporting their execution, in three aspects:

1. Conversion into runnable code (translation or interpretation)
2. leveraging on an execution engine (the library functionalities supporting the runnable code)
3. in turn relying on a software platform (JVM, .NET, and so on)

And the division of labour: the role of DSL engineers mostly focuses on steps 1 and 2, other than defining the syntax.

Two ways to convert a DSL into runnable code

In both cases there are technical prerequisites: a parser for the actual syntax of the DSL should exist or be generated, and the execution engine for the target platform should exist. The engine is unavoidable: be it internal or external, transpiled or interpreted, the DSL needs an execution engine — a library providing the functionalities of the DSL. This is no different from any other library supporting some given domain, except that hacks could be exploited to ease the adoption of the target DSL syntax.

7. External and internal DSLs

Everything discussed so far concerns external DSLs: languages whose syntax is totally custom, hence requiring a custom parser. The alternative is an internal (or embedded) DSL: the syntax is a subset of some pre-existing GPL, one whose syntax is flexible enough to allow customisation.

Creating internal DSLs is described as a recent trend, enabled by the wide adoption of flexible GPLs such as Kotlin, Groovy or Scala, which come with ad-hoc constructs: trailing-lambda convention, infix notation, operator overloading. Examples you may already know: the Kotlin DSL for Gradle, and SBT.

plugins {
    `java-library`
}

dependencies {
    api("junit:junit:4.13")
    implementation("junit:junit:4.13")
    testImplementation("junit:junit:4.13")
}

sourceSets {
    main {
        java.srcDir("src/core/java")
    }
}

java {
    sourceCompatibility = JavaVersion.VERSION_11
    targetCompatibility = JavaVersion.VERSION_11
}

The lecture reads that file as follows: the domain is build automation; this is pure Kotlin plus the Gradle library, which contains an execution engine; and the Gradle library is designed to be used as a Kotlin DSL.

Key aspects of internal DSLs

8. Tools for MDD, and Xtext

About Xtext

Xtext is a framework for MDD and, in particular, external DSLs. The characterisation given in the lecture is worth reading twice, because it collapses two activities that are usually separate:

• Xtext provides a language for defining languages…
• …which is also a meta-modelling language;
• therefore meta-modelling and DSL definition are done simultaneously.

From a single grammar file, Xtext automatically generates the full language infrastructure: model interfaces and classes (EMF compliant), parser, validator (with pluggable rules), transpiler stub, scoping (with pluggable rules), IDE support via LSP, syntax colouring, test stubs, and more.

9. The Sheduler running example

The running example is the task scheduling domain, and the language is called Sheduler — Sheduler ≡ Shell + Scheduler. The domain description, in the style of Chapter 2:

• Users may be willing to schedule custom tasks on a machine.
• Task ≡ running any command available on the OS, via some shell of choice (bash, cmd, powershell).
• Scheduling implies defining when the task should be executed: relatively to now (in 5 minutes, in 1 hour), absolutely (today at 10:00, tomorrow at 12:00, on 2023/11/23 at 13:16), before or after some other task, or periodically (every 5 minutes, every 48 hours).
• Note that tasks may be inter-dependent, because of the before/after relations.

The plan of work is stated up front: use Xtext's meta-modelling language to define the domain of task scheduling; simultaneously define the syntax of the DSL; add scoping and validation rules; design and implement the execution engine (exploiting Java's ScheduledExecutorService); and finally create a code generator producing Java code from the DSL.

Project structure

sheduler-lang/
├── build.gradle
├── gradle/
├── gradle.properties
├── gradlew
├── gradlew.bat
├── it.unibo.spe.mdd.sheduler/
│   ├── build.gradle
│   └── src/main/java/it/unibo/spe/mdd/sheduler/sheduler
│       ├── GenerateSheduler.mwe2
│       └── Sheduler.xtext
├── it.unibo.spe.mdd.sheduler.ide/
│   └── build.gradle
├── it.unibo.spe.mdd.sheduler.web/
│   └── build.gradle
└── settings.gradle

Relevant Gradle tasks

• generateXtextLanguage generates the language infrastructure (domain model interfaces and classes, parser, validator stub, scoping stub, code generation stub). It is automatically executed by Gradle before compilation; run it manually only to force regeneration.
• shadowJar generates the runnable jar for the LSP server; run manually to deploy it.
• jettyRun starts the web playground; run manually during manual testing of the language.
• Ordinary Gradle tasks for compilation and testing behave as usual.

The grammar

This is the heart of the chapter: a file that is simultaneously a meta-model and a syntax definition.

A program in the resulting language reads like this — paste it into the web playground started by jettyRun at http://localhost:8080, and press Ctrl+Space to see the auto-completion menu:

pool {
    schedule task greetWorldFrequently {
        command "echo hello"
        entry point "/bin/sh -c"
        in 5 minutes
        repeat every 1 hours
    }
}

From the grammar Xtext generates the model interfaces: TaskPoolSet, TaskPool, Task, AbsoluteTime, Date, ClockTime, RelativeTime, TimeSpan, the TimeUnit enum, and a ShedulerFactory with one creation method per type — and classes are generated too. Compare that list with Chapter 3: a factory plus a composition hierarchy of entities and value-like objects, obtained without writing a line of Java.

10. Validation and scoping rules

A grammar decides what can be parsed. It cannot decide what is meaningful: 25:99 parses perfectly as a ClockTime. That gap is filled by two pluggable mechanisms whose stubs Xtext generates for you.

Validation

Validation rules are defined in the ShedulerValidator class (package it.unibo.spe.mdd.sheduler.validation, Gradle sub-project sheduler-lang/it.unibo.spe.mdd.sheduler). The stub class is generated when running generateXtextLanguage, which simply triggers the execution of GenerateSheduler.mwe2.

package it.unibo.spe.mdd.sheduler.validation;
import it.unibo.spe.mdd.sheduler.sheduler.*;
import org.eclipse.xtext.validation.Check;
import org.eclipse.xtext.validation.CheckType;

public class ShedulerValidator extends AbstractShedulerValidator {
    @Check(CheckType.FAST)
    public void ensureDateIsValid(Date date) {
        if (date.getYear() < 0) {
            error("Year must be positive", date, ShedulerPackage.Literals.DATE__YEAR, 0);
        }
        if (date.getMonth() < 1 || date.getMonth() > 12) {
            error("Month must be between 1 and 12", date, ShedulerPackage.Literals.DATE__MONTH, 0);
        }
        if (date.getDay() < 1 || date.getDay() > 31) {
            error("Day must be between 1 and 31", date, ShedulerPackage.Literals.DATE__DAY, 0);
        }
    }
}

The remarks are the kind of detail that only matters when you sit in front of the IDE, which is exactly why they are worth memorising:

• The name of the validation method is meaningless: only the type of the parameter matters, plus the presence of the @Check annotation.
• The check type is optional and defaults to CheckType.NORMAL: FAST runs whenever a file is modified, NORMAL when saving the file, EXPENSIVE only when explicitly validating via the menu option.
• error(...) may be replaced by warning(...) for minor issues.

Exercise 1 — custom validation rules

Write custom validation rules covering the following constraints (utility methods are available in class TimeUtils):

• Warning if representing some RelativeTime as a java.time.Duration would overflow.
• Warning if representing some AbsoluteTime as a java.time.LocalDateTime would overflow.
• Warning if some AbsoluteTime is in the past or in the present (only the future is admitted) — see LocalDateTime.now() and LocalDateTime.isBefore().
• Error if some ClockTime is invalid: hour 0-23, minute 0-59, second 0-59, millisecond 0-999, nanosecond 0-999.
• Error if some TimeSpan is invalid: negative duration; 1000 or more when the unit is nanoseconds or milliseconds; 60 or more when the unit is minutes or seconds; 24 or more when the unit is hours.
• Error if any two tasks from the same pool have the same name.
• Error if any two pools have the same name.
• No periodicity should be allowed for tasks scheduled before/after some other task.

Scoping

Scoping rules are defined in the ShedulerScopeProvider class (package it.unibo.spe.mdd.sheduler.scoping), whose stub is likewise generated by generateXtextLanguage.

public class ShedulerScopeProvider extends AbstractShedulerScopeProvider {
    @Override
    public IScope getScope(EObject context, EReference reference) {
        return super.getScope(context, reference);
    }
}

• Only one method should be overridden.
• It should return an IScope object for each EObject containing some EReference — in this language, that can only happen in the before and after properties of tasks.
• The scope is essentially a container of EObjects, which are the possible values for the EReference, and the implementer selects which ones to include.

Exercise 2 — custom scoping rules

Write a custom scoping policy for the before and after properties of tasks, such that:

• Only tasks from the same TaskPool can be referenced by some task;
• Tasks from some pool cannot be referenced by tasks from other pools;
• Anonymous tasks (those without a name) cannot be referenced at all.

11. Execution engine, generator, interpreter

Giving semantics to the language means choosing one of the two approaches of §6 — interpretation or translation — and both require the definition of an execution engine, a library providing the functionalities of the DSL.

Designing the engine by asking what the platform already offers

The reasoning on the slide is a small masterclass in engineering economy:

• Is there functionality in the JDK which supports the scheduling of tasks in the future? If yes, use it; otherwise look for a third-party library, or implement it. Luckily, we may use ScheduledExecutorService.
• Same question for the execution of custom commands via some shell. Luckily, we may use ProcessBuilder.

The design insights follow: define a custom notion of ShedulerTask encapsulating the command, optionally the shell, the initial delay, optionally the period, optionally the tasks to be executed before/after, and the functionality for executing the command via ProcessBuilder; and a custom notion of ShedulerRuntime leveraging the ScheduledExecutorService API to schedule those tasks for execution.

public class ShedulerRuntime {
    private final ScheduledExecutorService delegate;

    public ShedulerRuntime(ScheduledExecutorService delegate) {
        this.delegate = Objects.requireNonNull(delegate);
    }

    public void schedule(SheduleTask task) {
        if (task.isPeriodic()) {
            delegate.scheduleWithFixedDelay(
                task.asRunnable(),
                task.getDelay().toMillis(),
                task.getPeriod().toMillis(),
                TimeUnit.MILLISECONDS
            );
        } else {
            delegate.schedule(
                task.asRunnable(),
                task.getDelay().toMillis(),
                TimeUnit.MILLISECONDS
            );
        }
    }
}

public Process executeAsync() throws IOException {
    return new ProcessBuilder(entrypoint, command).inheritIO().start();
}

public Runnable asRunnable() {
    return () -> {
        try {
            executeAsync();
        } catch (IOException e) {
            e.printStackTrace();
        }
    };
}

What generated code looks like

Given this DSL program:

pool myPool {
    schedule task greetFrequently {
        command "echo hello"
        entry point "/bin/sh -c"
        in 5 minutes
        repeat every 1 hours
    }
    schedule task greetOnce {
        command "echo hello"
        entry point "/bin/bash -c"
        at 2030/10/11 12:13
    }
}
pool otherPool {
    schedule task shutdownAfter1Day {
        command "sudo shutdown now"
        entry point "/bin/zsh -c"
        in 1 days
    }
}

the generator is expected to emit Java of this shape — note that each pool becomes a method, and that the relative and absolute times of the DSL have become Duration.parse and LocalDateTime.parse calls:

public static void main(String[] args) {
    ShedulerRuntime runtime = new ShedulerRuntime(Executors.newScheduledThreadPool(1));
    pool_myPool(runtime);
    pool_otherPool(runtime);
}

private static void pool_myPool(ShedulerRuntime runtime) {
    ShedulerTask task0 = ShedulerTask.in("greetFrequently", "echo hello", "/bin/sh -c", Duration.parse("PT5M"));
    task0.setPeriodic(Duration.parse("PT1H"));
    runtime.schedule(task0);
    ShedulerTask task1 = ShedulerTask.at("greetOnce", "echo hello", "/bin/bash -c", LocalDateTime.parse("2030-10-11T12:13"));
    runtime.schedule(task1);
}

Exercises 3, 4 and 5

Check your understanding