Improve speed of Machine Learning System (Engine) from 3 requests per second to 10000 requests per second
As a data Engineer, the data scientist team gave me a Machine Learning system with FastAPI for predicting probability of percentage that called Engine.
The engine is written in python using FastAPI and developed in docker system. by calling API and post patients data the engine sending back the percentage of probability of diseases.
Engine speed is very low, One of the biggest problems and the engine can handle only 3 requests per second while we predict at least have 2000 requests per second.
The nightmare of this problem is the engine result become in the first part of the application and it means that after the user registration and fill the questionnaire, the engine must answer the result immediately.
Post "https://host/engine/": dial tcp: host: no such
host{"time":"2021-03-15T05:34:26.350582953Z","level":"ERROR","prefix":"-","line":"288","message":"Engine Error Result : Post \"https://host/engine/\": dial tcp: host: no such host"}Using Kubernetes
At first, by increasing the number of instances in Kubernetes to 50 we reach 60 requests per second but it used about 66 percentage the CPU and ram of the cloud server.
Queue and Pub/Sub Dispatcher
One of my favorite way to queue process is using Pub-Sub Service and handling a million requests by queue jobs in Redis and sending with TCP protocol but using Pub/Sub can’t increase the number of requests per second.
Coding Theory
In my problem, the engine has a small State-space of about 4 million states so I’m coding input state into unique code as a key to get the response of the engine as value. so writing script Golang to make all states and sending them into the engine and save whole states as key-value in Redis. in this way I’m caching engine in Redis. this coding Theory isn’t the general solution but statistical cache base on most happening input can easily increase the number of requests per second.
Redis
Redis can easily be handling 10000 requests per second even more and finally write testing for 10000 requests per second and successfully pass the test.
$ redis-benchmark -t set,lpush -n 100000 -qSET: 74239.05 requests per secondLPUSH: 79239.30 requests
per second
Computing time
This algorithm make about 26 hours to cache the whole state in Redis.
Special Thanks
special thanks to Medi Mohammadi for handling the DevOps.
Visit us at DataDrivenInvestor.com
Subscribe to DDIntel here.
Join our creator ecosystem here.
DDI Official Telegram Channel: https://t.me/+tafUp6ecEys4YjQ1