Data management in Microservice Architecture pattern

Have you heard about Microservice Architecture pattern ?

Microservice Architecture pattern (MA pattern) is a pattern which decomposes your application into services. Then you can easily scale your application, easily maintain your application.
Each service work independently like a small application in your application.
MA pattern structures the application as the set of loosely coupled, collaborating services.

But, the more the loosely coupled services are, the more difficult to make data between services consistent.
In order to ensure loose coupling, each services should has it owns database. Now, you should think about the case : you have a transaction need to be done in 2 different databases. How do you do to ensure the transaction done ?

In this blog, I want to talk about the pattern which can help to solve this data consistent problem.

1. Shared database

Sound simple, right ?

We just use the same database for the several services. So the transaction can be guarantee done.

For example :
The online shop application have two microservices : Order and Customer.

If a order is finished, we need update status of order data and also reduce the amount of user's credit.

If we use technique Share database between Order service and Customer service. We can ensure the order data and customer data are consistent with each other by executing a transaction.

This technique give you a very simple way to implement. But sometime the Order service and Customer service are tight with each other. If we want to scale/ expand Order service, it may require you update the Customer service.

If your services are simple enough and depend on your application context, this technique can be a good choice for you. On the other hand, you should think about the other way.

Let go next for the other technique, it may help you in the complicated cases.

2. Saga

Preconditions : your microservices use different databases.

Problem: Data transactions

A saga is a sequence of local transactions.

Each local transaction updates the database and publishes an event to trigger the next local transactions in the saga.

If a local transaction fails then the saga executes a series of compensating transactions that undo the changes of last transaction.

The point of this pattern is : when you make a transaction to update the data in database, you also have to implement a sequence of queries that can undo the changes did by that transaction.

In some case, this implementation can make your application more complicated.

There are two ways of coordination saga:

a. Choreography-based saga: Each local transaction publishes event to trigger the local transaction in other service.

                                                              (source: microservices.io)

b. Orchestration-based saga: a service create a saga which tells what local transaction need to be executed.

                                                         

                                                        (source: microservices.io)

Result

This pattern enables your application to maintain data consistency across multiple services without using distributed transactions.

There are also the following issues to address:

• Service must  atomically updates its database.
• Application has to have a mechanism of publishing an event.

Let move to the next part of this blog to see how we can publish an event in our application. 

3. Event sourcing

Problem : Handle event of saga

In the previous section, we walks through Saga pattern which use event to trigger local transaction in other services to make the data in whole system consistence.
Let go deeper on how event work. How to make it reliable ? How to ensure the lossless when sending event to other services? How the state of data can be consistence by playing the events ?

Yes, Event sourcing pattern can help you to solve these issue. The core concept of Event Sourcing is:

•  Service persist the state of  business entity as a sequence of state-changing event.
• When service wants to reconstruct state of business entity, just replay the sequence of event.
• When service wants to undo a transaction, just make new event which do the opposite things from events of that transaction.

But,
Now you can imagine how hard you try to make a query to retrieve the current state of your order, user or filter your order, user by some conditions. Remember, in this pattern you do not store the current state of order, user, you store the list of events which change the state of order, user instead.
Well, it can be your bottle necks. You may have to spend a whole day for 1 query then you realize your query takes too much time for execution.
It's a big problem but let see how we can solve this problem by patterns below.

4. CQRS

Problem: Query data for presentation

It stand for Command Query Responsibility Segregation 

The idea of this pattern is separating your application into two parts :  Command part and Query part.

Command part : handle create, update, delete, rollback which emits event when data changes.

Query part: handle queries by executing them against one or more materialized views that are kept up to date by subscribing to the stream of events emitted when data changes.

Let image you have a event data source which store your events emitted by Command part. You also have another data source which subscribes to event data source, whenever event data source has new event, this data source will update it's data base on event payload.

The point is  your data source for query part has a-easy-for-query-structure.

Then whenever your application wants to query data it will query from query data source.

Okay, now you can solve the problem of query. But in order to implement this pattern you have to consider about :
- Complexity : your application have to separate into part, each operation need to specify the data source : event data source or query data source ?

- The process which query data source replicated from event data source can be lag, interrupted so your need to handle them.

5. API Composition

Problem: Retrieve data in a microservice architecture

Let take a look to example at the begining of this blog. Your application has two micro services :  Order and User.

You have a requirement : implement an api for getting both user and order. How do you solve it ?
We have API Composition.

Let start doing this requirement by making a API composer which invokes service's api which own the necessary data, then perform an in-memory join of the results from services' api.

Conclusion

Keep your application data consistency is very important. Measure data consistency can lead you to many problem like data source reliable, complexity, performance... So you often have to apply  more than one pattern to solve these problem.
But depends on your application, your current context, you can choose the most suitable pattern to apply to your application.  Sometime you can accept the high complexity of application to reach the higher performance but sometime you choose the opposite. It's both okay as long as you reach your goal and solve your problem. 

If you liked this article

Let's subscribe the updates of Scuti!

Share on Google Plus

About Anonymous

This is a short description in the author block about the author. You edit it by entering text in the "Biographical Info" field in the user admin panel.