My experience to build a CQRS system with microservices architecture in a financial system to handle 10k requests per second
In the last 6 months, I made enormous system for online payment and settlement with bank for handle 10k TPS. In this article I will explain about some details and problem in my project.
In this project, I faced many distinct challenges simultaneously. First, this system has to handle 10k requests per second to submit a payment in the Bank system. In addition, our third-party Bank system, unlike other financial systems, doesn’t have any kind of inquiry. It means that we have to handle any failed scenario. Furthermore, we need a worker pod with a very special mechanism to settle the payment at the midnight and it needs many hard systems to send batches at the special time for settlement in the Central Bank.
Microservices best solution for separate parts
We separate all unrelated parts of code into different pods. It means that all parts are in their pod, It really good part because after developing a prototype of the system we can easily assign different programmers to each part. also, we can increase the number of pods to handle more requests per second.
Database bottleneck problem
In our architecture, we need to handle 10k TPS in the payment pod. Because we make the financial system, store all requests in the persistence database. If any of the transactions occurred failed, we have to persistence log to find out what’s going on in our system and find what causes this failure. Finally, we have to answer our clients “where is the money?”.
In this situation, the database in the payment pod is engaged in reading and writing. The worst scenario is we need to group by query to make batch files and send them to settle. Also, we need to add a transaction for the update situation of the batch that happens in settle, furthermore, customers call our API to get clients’ updates.
If we keep all of these queries in one single database we will have a database bottleneck and the database in payment has serious problems in 10k TPS.
Because we face bottlenecks in the database, we separate the database of payments. In the Payment pod only insert in one table, and we don’t have any inner join or outer join. In addition, we use Redis to read data in payment. Conclude that in payment we have only one table and use only insert queries. On another hand, we send successful payments with a message broker into the settlement pod. In the settlement, we use group by and transaction to make baches and settle transactions. Furthermore, send failed and successful transactions to the report pod to make the report for customers and clients.
This method will give the powerful tools to handle more TPS without serious problems in the database.
One of the biggest disadvantages of the CQRS design pattern is a change in the database is so difficult, and if some changes in the database happened, we have to change many databases.
Does the CQRS Design pattern a good method for my problem?
CQRS separates reads and writes into different models, using commands to update data, and queries to read data.
CQRS Design Pattern
Normally, in monolithic applications, most of time we have 1 database and this database should respond both query and update operations. That means a database is both working for complex join queries, and also perform CRUD operations. But if the application goes more complex this query and crud operations will be also is going to be un-manageable situation.
When to use the CQRS pattern?
The CQRS pattern is useful in a few specific scenarios:
- Collaborative domains. The primary application of CQRS is in collaborative domains, where numerous users can view the same data concurrently.
- Task-based user interfaces. It’s also helpful with task-based user interfaces, where users are led through a series of steps to finish a challenging task.
- High-traffic systems. With the help of CQRS, better performance and scalability can be achieved by evenly distributing the workload between read and write operations.
- Querying from repositories. CQRS is useful when it’s challenging to query all the data that users need to view from repositories.
- Complex business logic. CQRS can assist in streamlining the design of an application if it contains complicated business logic by separating the read and write processes.
- Optimizing read operations. For applications that have a large number of read operations, CQRS can optimize those operations by creating dedicated read models.
- Different data models for reads and writes. In some instances, the data model used for writing to the database might not be appropriate for querying. Through CQRS, multiple data models can be established for each operation by separating the read and write models.
- Support for event sourcing. CQRS can be combined with event sourcing patterns to develop a system that can handle a large number of events and queries.
It’s crucial to remember that CQRS should only be used in restricted situations where it’s truly necessary. It’s useless for systems that follow the CRUD — create, read, update and delete — mental model.
Subscribe to DDIntel Here.
Visit our website here: https://www.datadriveninvestor.com
Join our network here: https://datadriveninvestor.com/collaborate