{
"slug": "Transform-database-bottleneck-in-software-into-the-CPU-problem-21859f9cafa0",
"title": "Transform database bottleneck in software into the CPU problem",
"subtitle": "We have an application with Microservice architecture. In normal situations the application transfers 10k requests per second. We have a…",
"excerpt": "We have an application with Microservice architecture. In normal situations the application transfers 10k requests per second. We have a…",
"date": "2023-08-28",
"tags": [
"My Experience",
"Microservices",
"Architecture"
],
"readingTime": "2 min",
"url": "https://medium.com/@mobinshaterian/transform-database-bottleneck-in-software-into-the-cpu-problem-21859f9cafa0",
"hero": "https://cdn-images-1.medium.com/max/800/1*vhAuMBGSYIpIcMssrB1qZA.png",
"content": [
{
"type": "heading",
"level": 2,
"text": "Transform database bottleneck in software into the CPU problem"
},
{
"type": "paragraph",
"html": "We have an application with Microservice architecture. In normal situations the application transfers 10k requests per second. We have a problem with contract tables. We cache all contracts in the Redis but had an IO bottleneck. In this article, I want to discuss different methods for changing the IO bottleneck into the CPU problem. It means using the decrypt and encrypt method to solve this problem."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*vhAuMBGSYIpIcMssrB1qZA.png",
"alt": "Transform database bottleneck in software into the CPU problem",
"caption": "",
"width": 723,
"height": 301
},
{
"type": "paragraph",
"html": "In this project, we have important contracts for each business. every user based on the client business, enrolls in our system and fills the base on the contract."
},
{
"type": "paragraph",
"html": "Our backend part, we have six different microservices. that has its own database. and we pass contract uuid to all of these microservice and get value of contract from one microservice."
},
{
"type": "paragraph",
"html": "In our back-end part, we have six different microservices. That has its own database. And we pass the contract uuid to all of these microservices and get the value of the contract from one contract microservice."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*xuGecssVc2n3MtTiDpXL9Q.png",
"alt": "Transform database bottleneck in software into the CPU problem",
"caption": "",
"width": 780,
"height": 341
},
{
"type": "paragraph",
"html": "Suppose that we have 10k requests per second and all six microservices get data from the contract table. Now we have 60K requests per second on the contract table."
},
{
"type": "code",
"lang": "text",
"code": "10K * 6 = 60K (TPS)"
},
{
"type": "paragraph",
"html": "In normal situations, Redis can handle 10K TPS easily. DevOps can increase this value to 60K with an increased number of Redis instances."
},
{
"type": "heading",
"level": 2,
"text": "Get once and send data into other Microservices"
},
{
"type": "paragraph",
"html": "In the first Method, I can get data from the service one and send contract data with GRPC to other microservices. Now the contract table is not a bottleneck at all. But If the contract table changes, I have to change data in many GRPC flows. And if the employer needs to change the contract several times, it will be thoughtful."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*9JQlAaTHFdqj_vo8RML3pw.png",
"alt": "Transform database bottleneck in software into the CPU problem",
"caption": "",
"width": 820,
"height": 351
},
{
"type": "heading",
"level": 2,
"text": "Encrypted contract information"
},
{
"type": "paragraph",
"html": "In this method, I encrypt the data of the business contract and give it to the user. A user sent us encrypted data of business. All six microservices decrypted the data and used them. So we don’t have any pressure on the contract table. Decrypted data needs CPU usage. It means that we transform the IO bottleneck into a CPU problem."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*DFTb8BhNvfXYUnOWKeaSIA.png",
"alt": "Transform database bottleneck in software into the CPU problem",
"caption": "",
"width": 826,
"height": 322
},
{
"type": "heading",
"level": 2,
"text": "Add hash key for contract data in front end and use decode and encode contract"
},
{
"type": "paragraph",
"html": "In this method, first, make a hash code for hashcode information and then just encode and decode the contract information in our system."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*9JQlAaTHFdqj_vo8RML3pw.png",
"alt": "instead of UUID use hashcode",
"caption": "instead of UUID use hashcode",
"width": 820,
"height": 351
},
{
"type": "paragraph",
"html": "Contract Information encodes data into one single field."
},
{
"type": "paragraph",
"html": "Special thanks to Majid.Mohsenifa and Mohsen Mottaghi"
},
{
"type": "paragraph",
"html": "Subscribe to DDIntel Here."
},
{
"type": "paragraph",
"html": "DDIntel captures the more notable pieces from our main site and our popular DDI Medium publication. Check us out for more insightful work from our community."
},
{
"type": "paragraph",
"html": "Register on AItoolverse (alpha) to get 50 DDINs"
},
{
"type": "paragraph",
"html": "Support DDI AI Art Series: https://heartq.net/collections/ddi-ai-art-series"
},
{
"type": "paragraph",
"html": "Join our network here: https://datadriveninvestor.com/collaborate"
},
{
"type": "paragraph",
"html": "Follow us on LinkedIn, Twitter, YouTube, and Facebook."
}
]
}