Hi Today I am going to discuss about how to develop Event-driven microservices using Quebic framework . If you are new to microservices you can refer my previous article which covered main concepts about microservices. And If you are new to Quebic please go through with this documents.
Main Topics
- Why we need Event-driven Microservices
- Event Sourcing
- Event-driven
- CQRS
- Sample Application - Event driven coffee shop
- Events flow
- Getting started with Quebic
- Quebic - EventBox
- Implementation
- How to run sample application
1. Why we need Event-driven Microservices
No Foreign key constrain between services
Assume you have two services. One service user-service and other one is search-service that service is optimized for searching. When user-service stores new user-data it also call search-service to store related data. If someone delete user-data from user-service , The search-service don’t know about that deletion.
Distributed transactions are hard
When some transaction is happened through the multiple services, If some issue occur within the execution , Since transaction touched multiple places then hard to identify who is the responsible for the reproduce/rollback.
High coupling between services
When we want to exchange information between services we need to know about each others. Then needs to know how to reach another service and how to response it.
Hard to scale
In the traditional Microservices architecture all the inter-services communication are happen request-response way. If one service is slow all the caller stack of that service getting slow. That’s make hard to scale the system.
Lets see how we can use Event Sourcing and Event-driven architecture concepts to minimize these challenges.
2. Event Sourcing
Event Sourcing stores every state transactions as an event in a log.
All the events are stored for a particular domain. That domain-events are the main key concept behind in the Event Sourcing architecture.
The domain-event is a data structure that representation a some thing happen in the past. Domain-events are immutable and immortal. That’s mean never change or never remove it from Event-Store.
With Event Sourcing architecture, we have an audit-trail to tell how things happen in the past. Now we know how to reproduce / rollback the state of the system. This is really help to identify issues occurs in distributed systems.
3. Event-driven
The Event-driven architecture helps to remove the high coupling between services. Each services can run independently and no need to know about how to reach others. Every one listening for there own events.
With Event-driven Microservices, all the operations are happened in eventually consistence way. Instead of waiting and get the response, user get tracker to track his request. But eventually he can get the correct response.
Lets see how to implement Event-driven Microservices using these concepts.
4. CQRS
CQRS stands for Command Query Responsibility Segregation. This pattern is used to create materialized views from streams of events. CQRS pattern is built based on Event-sourcing and Event-driven architecture concepts.
According to the CQRS pattern , you can divide your application’s read and write parts into separate modules. Read part is called query model and write part is called command model.
But it is not required to keep command and query models into separate projects. If you want you can keep both modules in a single project then those module are running on same process. But those module are fully independent, any time you can separate those.
CQRS separates reads from writes. Then it helps to scale both parts independently. Each sides are optimized.
OK I think, we have covered most of the theories. Lets jump into implementation.
5. Sample Application - Event driven coffee shop
Here you can see we have three Microservices. orders-service, beans-service and barista-service. Source location.
orders-service is responsible for manage orders which coming from users.
beans-service is responsible for storing and validating beans before process the order.
barista-service is responsible for making coffee according to the order.
6. Events flow
I am not going to explain whole event flow, but I will explain some of the important points. Here first step is orders-service is listening for request.OrderPlaced event.
That event is fired by API-Gateway. After orders-service is received the event, then it creates reference-id and response back that reference-id to the API-Gateway. Then user can use that reference-id to track his order eventually.
Then order-service invokes the placeOrder() method. That method is responsible for creating orders.OrderPlaced domain-event. The reference-id ( created in previous step ) is used as the aggregate-id of the domain-event.
7. Getting started with Quebic
Quebic is a framework for writing serverless functions to run on Kubernetes. You can write your functions in any language. Currently Quebic supports only for Java and NodeJS. Github repo.
8. Quebic - EventBox
EventBox enables the Event-Sourcing capability of the Quebic. With Eventbox, you can maintain Event-store for your domains and EventBox provides way to inspect domain-events under particular domains. That makes help to get advantages of Event-Sourcing into your applications. You can access EventBox by using its REST-Interface or Dashboard UI.
Create Domain
- Dashboard UI
Go to the Event Store section and then click create button. Then it will open Domain create interface. There are three mandatory field.
Name : Domain name. Provide unique name for your domain for the identification
Event Group : This will be the prefix value of the event types. Lets take some events types.
Eg: orders.OrderPlaced, orders.OrderBeanValidated
Here all the events types are started with ‘orders’ prefix. That is the event group of that event. By mentioning event-group, This domain is going to listening only for the orders.* event types.
Aggregate ID Field : Aggregate field name of the domain-event. You can set the identity field name of the domain object.
- REST-Interface
curl -X POST http://eventbox-host:port/api/domains-H 'content-type: application/json'-d '{"name":"orders","eventGroup":"orders","aggregateIdField":"id"}'
Fetch Domain-Events
- Dashboard UI
Go to the Event Store section and It will display created domains. Open a right-menu from one of that domains. Select Domain-Event link of the menu. Then it will be redirected to the domain-events pages.
- REST-Interface
curl http://eventbox-host:port/api/domains/<domain-name>/domain-events
9. Implementation
According our sample project we have three Microservices orders-service, beans-service and barista-service. We can divided orders-service into two modules orders-commond-service and orders-query-service according to the CQRS pattern. We are going to implement each modules using Quebic functions.
Here I am going to explain some important points. You can find whole implementation to this sample application in here.
orders-command-service
In orders-commond-service, users are allowed only for write operations. This is a part of the it’s function-spec file.
...replicas: 1events:
- request.OrderPlaced
- orders.OrderBeanValidated
- orders.OrderBeanCancelled
- orders.CoffeeBrewStarted
- orders.CoffeeBrewFinished
According to the events section, this function is going to listening only for request.OrderPlaced, orders.OrderBeanValidated, etc.. events.
Let’s check its handler implementation. It use context.eventID for mapping request into particular methods.
orders-query-service
In orders-query-service, users are allowed only for read operations.
When receive an event which start from ‘orders’ ( eg: orders.OrderPlaced ) prefix, Then the event’s payload is stored in the query data-store. For this example I just used JavaScript data structure as the query data-store. But if you want you can use some external database for query data-store.
Let’s check its handler implementation. You can see, Its call fetchLatestPayloads() method when loading function-container. Inside that It makes a call for the EventBox to get latest event’s payloads for a particular domain.
http://eventbox-host:port/api/domains/<domain-name>/domain-events/latest-payloads
EventBox stores all domain events, This endpoint represent the current state of the system under particular domain.
When orders-query-service start/re-start it invokes fetchLatestPayloads() method. This will help to up to date query data-store with domain-event-store and it make sure all the instances of the orders-query-service keep same query data-store.
You can get the Connection URI of the EventBox using EVENTBOX_URI environment variable.
Routing rules
let’s take a look at order-place routing rule. This is a sample http-request for placing new order. This will return created order with order-id.
api-gateway-host:port/orders POST
Content-Type: "application/json"{"beanType":"bean_type_1"}
let’s take a look at order-fetch routing rule. This is a sample http-request to fetch order details.
api-gateway-host:port/orders/{order-id} GET
After user send the http-request to API-Gateway then it fires request.OrderFetch event. The orders-query-service is received that event then get the relevant order details from the query data-store.
This order-id is the reference-id which created when placing the order. And this order-id is the aggregate-id of the domain event.
10. How to run sample application
This document contains all instructions to run sample application.
I think you got a considerable knowledge about Event-driven Microservices and how to implement it with Quebic. Please contact me if there are any problems. I really expecting your valuable feedback. Thanks for reading. Good luck.