{
"slug": "Improve-speed-of-Machine-Learning-API--Engine--from-3-requests-per-second-to-10000-requests-per--4d649d246a99",
"title": "Improve speed of Machine Learning API (Engine) from 3 requests per second to 10000 requests per…",
"subtitle": "as a data Engineer, the data scientist team gave me API Machine Learning called Engine.",
"excerpt": "as a data Engineer, the data scientist team gave me API Machine Learning called Engine.",
"date": "2021-03-23",
"tags": [
"My Experience",
"API",
"Performance",
"Data Engineering",
"Machine Learning"
],
"readingTime": "2 min",
"url": "https://medium.com/@mobinshaterian/improve-speed-of-machine-learning-api-engine-from-3-requests-per-second-to-10000-requests-per-4d649d246a99",
"hero": "https://cdn-images-1.medium.com/max/800/1*88coeBMjhI5q1p5WSpZoWg.jpeg",
"content": [
{
"type": "heading",
"level": 2,
"text": "Improve speed of Machine Learning System (Engine) from 3 requests per second to 10000 requests per second"
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*88coeBMjhI5q1p5WSpZoWg.jpeg",
"alt": "Improve speed of Machine Learning API (Engine) from 3 requests per second to 10000 requests per…",
"caption": "",
"width": 745,
"height": 483
},
{
"type": "paragraph",
"html": "As a data Engineer, the data scientist team gave me a Machine Learning system with FastAPI for predicting probability of percentage that called Engine."
},
{
"type": "paragraph",
"html": "The engine is written in python using FastAPI and developed in docker system. by calling API and post patients data the engine sending back the percentage of probability of diseases."
},
{
"type": "paragraph",
"html": "Engine speed is very low, One of the biggest problems and the engine can handle only 3 requests per second while we predict at least have 2000 requests per second."
},
{
"type": "paragraph",
"html": "The nightmare of this problem is the engine result become in the first part of the application and it means that after the user registration and fill the questionnaire, the engine must answer the result immediately."
},
{
"type": "code",
"lang": "text",
"code": "Post \"https://host/engine/\": dial tcp: host: no such\nhost{\"time\":\"2021-03-15T05:34:26.350582953Z\",\"level\":\"ERROR\",\"prefix\":\"-\",\"line\":\"288\",\"message\":\"Engine Error Result : Post \\\"https://host/engine/\\\": dial tcp: host: no such host\"}"
},
{
"type": "heading",
"level": 2,
"text": "Using Kubernetes"
},
{
"type": "paragraph",
"html": "At first, by increasing the number of instances in Kubernetes to 50 we reach 60 requests per second but it used about 66 percentage the CPU and ram of the cloud server."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*RyIo6zl5GEI7h8kXZ_JFEw.jpeg",
"alt": "Improve speed of Machine Learning API (Engine) from 3 requests per second to 10000 requests per…",
"caption": "",
"width": 885,
"height": 443
},
{
"type": "heading",
"level": 2,
"text": "Queue and Pub/Sub Dispatcher"
},
{
"type": "paragraph",
"html": "One of my favorite way to queue process is using Pub-Sub Service and handling a million requests by queue jobs in Redis and sending with TCP protocol but using Pub/Sub can’t increase the number of requests per second."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*Yf89N3LBlaoC4-E0q7tVTA.jpeg",
"alt": "From Pub-Sub pattern to Observer pattern",
"caption": "From Pub-Sub pattern to Observer pattern",
"width": 512,
"height": 406
},
{
"type": "heading",
"level": 2,
"text": "Coding Theory"
},
{
"type": "paragraph",
"html": "In my problem, the engine has a small State-space of about 4 million states so I’m coding input state into unique code as a key to get the response of the engine as value. so writing script Golang to make all states and sending them into the engine and save whole states as key-value in Redis. in this way I’m caching engine in Redis. this coding Theory isn’t the general solution but statistical cache base on most happening input can easily increase the number of requests per second."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*qI5U3oGIVdP9u6RpIsx_og.jpeg",
"alt": "Improve speed of Machine Learning API (Engine) from 3 requests per second to 10000 requests per…",
"caption": "",
"width": 1080,
"height": 256
},
{
"type": "heading",
"level": 2,
"text": "Redis"
},
{
"type": "paragraph",
"html": "Redis can easily be handling 10000 requests per second even more and finally write testing for 10000 requests per second and successfully pass the test."
},
{
"type": "code",
"lang": "text",
"code": "$ redis-benchmark -t set,lpush -n 100000 -qSET: 74239.05 requests per secondLPUSH: 79239.30 requests\nper second"
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*u8lSUfgCpD7IafLCOkvyDQ.png",
"alt": "Improve speed of Machine Learning API (Engine) from 3 requests per second to 10000 requests per…",
"caption": "",
"width": 1920,
"height": 641
},
{
"type": "heading",
"level": 2,
"text": "Computing time"
},
{
"type": "paragraph",
"html": "This algorithm make about 26 hours to cache the whole state in Redis."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*JelipCv_t4FconmbMvERSA.jpeg",
"alt": "Improve speed of Machine Learning API (Engine) from 3 requests per second to 10000 requests per…",
"caption": "",
"width": 1080,
"height": 251
},
{
"type": "heading",
"level": 2,
"text": "Special Thanks"
},
{
"type": "paragraph",
"html": "special thanks to Medi Mohammadi for handling the DevOps."
},
{
"type": "paragraph",
"html": "Visit us at DataDrivenInvestor.com"
},
{
"type": "paragraph",
"html": "Subscribe to DDIntel here."
},
{
"type": "paragraph",
"html": "Join our creator ecosystem here."
},
{
"type": "paragraph",
"html": "DDI Official Telegram Channel: https://t.me/+tafUp6ecEys4YjQ1"
},
{
"type": "paragraph",
"html": "Follow us on LinkedIn, Twitter, YouTube, and Facebook."
}
]
}