Breaking up with Flask and FastAPI: Why they don’t scale for ML model serving

Re:Breaking up with Flask and FastAPI: Why they don’t scale ... by kroolik on Thursday June 16, 2022 @04:58PM (#31770898)

Would you describe getting response from the model within few milliseconds cpu or memory intensive? What I assumed as to the characteristics of inference is a process that takes multiple seconds to minutes.

In a system that requires a response to the customer within a few milliseconds through web api, how do you ensure the performance in Python? I'm genuinely interested as that sounds outside of what stock CPython can do except trivial logic.

Another aspect that's completely ignored are the requirement - you might need the response time in milliseconds, couple of times per day. Others might need to serve hundreds of requests. You also need to consider the budget available.

[ Parent | on HN ]

Re:Breaking up with Flask and FastAPI: Why they don’t scale ... by hetspookjee on Friday June 17, 2022 @12:18AM (#31774078)

Just to add some ballparks: I have a yolov5 model that I run in PyTorch on a 3080 and the inference time is roughly 20ms. And inference with a 200estimator random forest is a few ms on a cpu. I can imagine that in some scenarios it’s not enough though.

[ Parent | on HN ]

Re:Breaking up with Flask and FastAPI: Why they don’t scale ... by kroolik on Friday June 17, 2022 @09:49AM (#31778113)

Indeed it might not be enough, and there is no one size that fits them all ;) For inference times of 50ms I wouldn't bother complicating the setup that much. Again, it all depends on your requirement and trade offs available.

[ Parent | on HN ]