{ "slug": "I-struggle-to-read-and-write-100K-requests-in-Postgres-DB--and-my-aggressive-solution-with-Redis--91461a8316a1", "title": "I struggle to read and write 100K requests in Postgres DB, and my aggressive solution with Redis.", "subtitle": "I faced 100K requests in a short period in our application. In that situation, Postgres Database has a lot of delays in response requests…", "excerpt": "I faced 100K requests in a short period in our application. In that situation, Postgres Database has a lot of delays in response requests…", "date": "2022-08-15", "tags": [ "Star", "Redis", "Postgres" ], "readingTime": "3 min", "url": "https://medium.com/@mobinshaterian/i-struggle-to-read-and-write-100k-requests-in-postgres-db-with-my-aggressive-solution-with-redis-91461a8316a1", "hero": "https://cdn-images-1.medium.com/max/800/1*FuRIfaWhy_wMrugAoL13ig.png", "content": [ { "type": "heading", "level": 2, "text": "I struggle to read and write 100K requests in Postgres DB, and my aggressive solution with Redis." }, { "type": "paragraph", "html": "I faced 100K requests in a short period in our application. In that situation, Postgres Database has a lot of delays in response requests, and response time enormously increased from about 1ms to 30 seconds, so the customer can’t open this Microservice on our application. My serious problem is that we will have significant competition in front of us, and maybe we will face more than 100K requests in a short time." }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*FuRIfaWhy_wMrugAoL13ig.png", "alt": "https://www.percona.com/blog/2017/01/06/millions-queries-per-second-postgresql-and-mysql-peaceful-battle-at-modern-demanding-workloads/", "caption": "https://www.percona.com/blog/2017/01/06/millions-queries-per-second-postgresql-and-mysql-peaceful-battle-at-modern-demanding-workloads/", "width": 763, "height": 470 }, { "type": "paragraph", "html": "I need to explain that our tables didn’t have any relation and only have two tables that store about 200 million records. We only use select and insert queries without any type of joining, and group between tables" }, { "type": "paragraph", "html": "Unfortunately, our Frontend part can’t cache votes because of security problems. Furthermore, we have many different types of Frontend that become impossible to cache simultaneously." }, { "type": "paragraph", "html": "Another simple work is not showing the previous votes of users, but the business layer has emphasized that we need this feature." }, { "type": "paragraph", "html": "As an immediate solution, our DB team investigates queries and correct table indexing. In another solution, I added the Redis part to store each query. Still, because every query related to user votes, it needs to cache for every user and uses a lot of RAM. Because of that, I can store it in Redis for only about 1 hour, and It also needs 1 query to get data from Database." }, { "type": "heading", "level": 2, "text": "Redis always be the solution." }, { "type": "paragraph", "html": "If somebody asks me about the best and most reliable tools to handle a million requests per second, I will say that Redis is the best tool. So defiantly, I want to use Redis instead of Postgres, but we have a painful problem: we don’t have enough Ram to store a million users in Redis." }, { "type": "paragraph", "html": "So what should we do?" }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*cnFEnd1yaZjMb1nF4aqphw.png", "alt": "https://redis.com/", "caption": "https://redis.com/", "width": 232, "height": 71 }, { "type": "heading", "level": 2, "text": "Cache immediate users" }, { "type": "paragraph", "html": "The first policy of cache is to say that if some user selects their votes right now, cache their value right now and keep them for about 1 hour." }, { "type": "paragraph", "html": "So if 8 million users come to our application and every vote size is about 1K, and every customer enroll 5 votes as a result that we need 40 GB RAM." }, { "type": "code", "lang": "text", "code": "8M * 1K *5 = 40GB" }, { "type": "paragraph", "html": "pros: straightforward implement
cons: minimum of one request needed for all users, we need 40GB RAM" }, { "type": "heading", "level": 2, "text": "Cache Exists Vote to recognize the new customer." }, { "type": "paragraph", "html": "In this policy, I want to recognize new and old users straightforwardly. Therefore, before starting television advertisements, read all data from the backup database to make a list of users that vote for a special event and store it as a list in the Redis." }, { "type": "code", "lang": "text", "code": "16 byte * 100M = 1.6GB" }, { "type": "paragraph", "html": "pros: only 1.6 GB RAM, no cost for new users" }, { "type": "paragraph", "html": "cons: calculate before Advertisement" }, { "type": "heading", "level": 2, "text": "Cache active users" }, { "type": "paragraph", "html": "In this policy, instead of storing all user’s votes, we select active users before the advertisement and hold all the votes of these users into Redis. It means that we keep only 100K import users that participated in the three last events. Based on this policy, we need to read data from the backup database before the advertisement and select 100k users and the probability that users will participate in future voting. We predict which of our old customers will join in future voting." }, { "type": "code", "lang": "text", "code": "100K * 1K *5 = 0.5GB" }, { "type": "paragraph", "html": "pros: only 0.5 GB RAM, predict active users" }, { "type": "paragraph", "html": "cons: calculate before Advertisement" }, { "type": "heading", "level": 2, "text": "Insert" }, { "type": "paragraph", "html": "In the other part, we have a very straightforward way of inserting. We can use Kafka to store the massive number of inserts and then slowly add them to our database. As I said, if immediate cache users do so, we have 1 hour to insert data into Posgtre, which is a lot of time." }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*lT3KlFYPL2j2q_AJzalTdQ.png", "alt": "https://kafka.apache.org/", "caption": "https://kafka.apache.org/", "width": 555, "height": 263 }, { "type": "heading", "level": 2, "text": "conclusion" }, { "type": "paragraph", "html": "If we combine these three methods, we need 42GB RAM and also need to preprocess investigate data, but for instance, use of the database, we use the cache, and it certainly increases the speed of request and response." }, { "type": "paragraph", "html": "Special thanks to Emad ghaffari" }, { "type": "paragraph", "html": "Subscribe to DDIntel Here." }, { "type": "paragraph", "html": "Join our network here: https://datadriveninvestor.com/collaborate" } ] }