Designing Event-Driven Microservices

1. Event-driven microservices: consumers and producers of event streams

An event-driven microservice is a microservice designed to interact by means of event streams — and it can be both a consumer of one set of input event streams and a producer of another set of output event streams. Consumer microservices consume and process events from one or more input event streams; producer microservices produce events to output event streams for other services to consume. The event streams are served by an event broker, and communication between event-driven microservices is completely asynchronous.

An event-driven microservice may be stateless or stateful, and it may also contain synchronous request–response APIs alongside its event-driven interfaces — the two styles coexist in the same service and in the same business topology.

2. Events and the two topologies

An event can be anything that has happened within the scope of the business communication structure — domain events: receiving an invoice, booking a meeting room, requesting a cup of coffee, hiring a new employee, successfully completing arbitrary code. Once these events start being captured, event-driven systems can be created to harness and use them across the organization. An event is a recording of what happened, much like how an application’s information and error logs record what takes place in the application — but differently from logs, which are technical, events are considered here the single source of truth, and as such they must contain all the information required to accurately describe what happened.

The topology is an important concept in event-driven microservices, and there are two kinds:

• the microservice topology — the event-driven topology internal to a single microservice: it defines the data-driven operations to be performed on incoming events, including transformation, storage and emission;
• the business topology — the graph-like relationship between individual microservices, event streams and request–response APIs that fulfils complex business functions.

3. Designing events: the truth pattern, singular definitions and event types

Patterns to support event identification and definition:

• “Tell the Truth, the Whole Truth, and Nothing but the Truth” — a good event definition is not simply a message indicating that something happened, but the complete description of everything that happened during that event: in business terms, the resultant data produced when input data is ingested and the business logic is applied. The output event must be treated as the single source of truth and recorded as an immutable fact for consumption by downstream consumers — consumers should not need to consult any other source of data to know that such an event took place.
• Use a Singular Event Definition per Stream — an event stream should contain events representing a single logical event. Mixing different types of events within one stream muddles the definitions of what the event is and what the stream represents.

Events are typically represented using a key/value format: the value stores the complete details of the event, the key is used for identification purposes (routing and aggregation operations on events with the same key), and the key is not a required field for all event types. There are three main event types:

• Unkeyed Event — a singular statement of fact with no key, e.g. an event indicating that a customer interacted with a product (a user opening a book entity on a digital book platform);
• Entity Event — describes the properties and state of an entity at a given point in time, where an entity is a unique business thing keyed on its unique ID — e.g. for a book publisher, a book entity keyed on ISBN;
• Keyed Event — contains a key but does not represent an entity; usually used for partitioning the stream of events to guarantee data locality within a single partition — e.g. a stream of events keyed on ISBN indicating which user has interacted with the book.

4. Materializing state from entity events

Entity events are particularly important in event-driven architectures: they provide a continual history of the state of an entity and can be used to materialize the state — only the latest entity event is needed to determine the current state of an entity. A stateful table can be materialized by applying entity events, in order, from an entity event stream: each entity event is upserted into the key/value table (inserting a new row if it doesn’t already exist, or updating it if it does), so that the most recently read event for a given key is represented.

Conversely, a table can be converted back into a stream of entity events by publishing each update to the event stream — this is the table–stream duality property, fundamental to the creation of state in an event-driven microservice. The two directions together close the loop that keeps streams and materialized state consistent.

5. Event data contracts and schemas

Event-driven microservices make event data important in two roles: as a means of long-term and implementation-agnostic data storage, and as a means of communication between services. Therefore producers and consumers must have a common understanding of the meaning of the data — ideally, the consumer must be able to interpret the contents and meaning of an event without consulting the owner of the producing service. This requires a common language for communication between producers and consumers: event-driven data contracts, analogous to an API definition between synchronous request–response services.

A well-defined data contract has two components:

• the data definition — what will be produced: the fields, types and data structures, i.e. defining an event schema;
• the triggering logic — why it is produced: the specific business logic that triggered the event’s creation.

The way to enforce data contracts and provide consistency is to define a schema for each event: the producer defines an explicit schema detailing the data definition and the triggering logic, with all events of the same type adhering to this format — a mechanism for communicating its event format to all prospective consumers, against which consumers can confidently build their business logic.

The schema format must support a full range of schema evolution rules: schema evolution enables producers to update their service’s output format while allowing consumers to continue consuming the events uninterrupted, and an explicit set of rules (the compatibility types: forward, backward, full) goes a long way in enabling consumers and producers to update their applications in their own time. Technologies such as Apache Avro and Google’s Protobuf provide schematization with an evolution framework — where certain sets of changes can be safely made without requiring downstream consumers to make a code change — and language-specific serialisations that generate typed classes to convert the schematized data into plain old objects in different languages.

6. The microservice single writer principle

An important principle in designing event streams:

Each event stream has one and only one producing microservice; this microservice is the owner of each event produced to that stream.

The outcomes of the single writer principle:

• the authoritative source of truth is always known for any given event, permitting the tracing of data lineage through the system;
• access control mechanisms should be used to enforce ownership and write boundaries on the streams.

7. The event broker: features, storage and the contrast with message brokers

At the heart of every production-ready event-driven microservices platform is the event broker: the system that receives events, stores them in a queue or partitioned event stream, and provides them for consumption by other processes. Events are typically published to different streams based on their underlying logical meaning. Event broker systems suitable for large-scale enterprises follow the same model: multiple, distributed event brokers work together in a cluster to provide a platform for the production and consumption of event streams.

Event broker features:

• Scalability — additional broker instances can be added to increase the cluster’s production, consumption and data storage capacity;
• Durability — event data is replicated between nodes, so a cluster can preserve and continue serving data when a broker fails;
• High availability — clients can connect to other nodes in the case of a broker failure, maintaining full uptime;
• High performance — multiple broker nodes share the load, and each node must handle hundreds of thousands of writes or reads per second.

Data storage features:

• Partitioning — event streams can be partitioned into individual substreams, so multiple instances of a consumer can process each substream in parallel, allowing far greater throughput;
• Strict ordering — data in a partition is strictly ordered and served in the exact order it was published;
• Immutability — all event data is completely immutable once published; previous data can be altered only by publishing a new event with the updated data;
• Indexing — events are assigned an index when written; consumers use it to manage consumption by specifying which offset to begin reading from, and the difference between a consumer’s current index and the tail index is the consumer lag, a metric used to scale consumers up or down;
• Infinite retention — event streams must be able to retain events in principle for an infinite period of time, which is foundational for maintaining state in an event stream;
• Replayability — streams must be re-playable so any consumer can read whatever data it requires: the basis for the single source of truth and for communicating state between microservices.

Event brokers vs message brokers. Event brokers can be used in place of a message broker — but a message broker cannot fulfil all the functions of an event broker. Message brokers enable systems to communicate through publish/subscribe message queues: producers write messages to a queue, a consumer consumes and processes them, and messages are acknowledged as consumed and deleted either immediately or shortly thereafter. Event brokers are designed around providing an ordered log of facts, which requires more complex capabilities. Two specific needs of event brokers are not satisfied by message brokers:

1. the message broker provides only queues of messages, where consumption is handled on a per-queue basis — applications that share consumption from a queue each receive only a subset of the records, making it impossible to correctly communicate state via events since each consumer cannot obtain a full copy of all events; the event broker instead maintains a single ledger of records and manages individual access via indices, so each independent consumer can access all required events;
2. a message broker deletes events after acknowledgment, whereas an event broker retains them for as long as the organization needs — deletion after consumption makes the message broker insufficient for providing the indefinitely stored, replayable, single source of truth for all applications; the event broker’s immutable, append-only log of facts lets any consumer pick up and reprocess from anywhere in the log at any time.

8. Single source of truth, the event loop and data liberation

The durable and immutable log provides the storage mechanism for the single source of truth: the event broker becomes the only location in which services consume and produce data, and every consumer is guaranteed an identical copy of the data. Adopting the event broker as the single source of truth requires a culture shift in the organization:

• no more direct SQL queries to access data in a monolith’s database — data must be published to the event broker, and any disagreement between the event streams and the monolith’s database is considered a failure;
• consumers of the data no longer couple directly on the monolith, but instead consume directly from the event streams;
• services can use only the event broker to store and access data — local copies of events may be used by the business logic, but the event broker remains the single source of truth for all data.

Most event-driven microservices follow, at a minimum, the same three steps — consume an event from an input event stream, process that event, and produce any necessary output events — starting a loop to poll the consumer client for new events and emitting any required output events. Some services derive their input event from a synchronous request–response interaction instead; in stream-sourced services the instance creates a producer client and a consumer client and registers itself with any necessary consumer groups.

Data liberation is the identification and publication of cross-domain data sets to their corresponding event streams — part of a migration strategy for event-driven architectures. Transitioning an organization requires integrating existing systems: migrating means making the necessary business domain data available in the event broker consumable as event streams, sourcing the data from the existing systems and state stores that contain it. Data liberation enforces the two primary features of event-driven architecture — the single source of truth and the elimination of direct coupling between systems — and the liberated event streams allow new event-driven microservices to be built as consumers, with existing systems migrated in due time. One method of liberating data uses a dedicated, centralized framework to extract data into event streams — e.g. Kafka Connect (exclusively for the Kafka platform), Apache Gobblin and Apache NiFi — each allowing a query against the underlying data set with the results piped through to the output event streams. Available event-driven microservices frameworks include Apache Kafka and the Confluent Platform, Spring Event-Driven and the Axon Framework.

Check your understanding