timliu99

> While there are several different methods to use your own executor pools or potentially use shared memory for a large model, all of these solutions are not first-class solutions for ML use cases...

Definitely not FastAPI's fault and yes Starlette has the same limitations. BentoML builds additional ML features/abstractions on top of Starlette. We introduced a "runner" concept which automatically creates separate processes for models to run in.

Disclaimer: I'm the author.

We've been using Flask for years as the foundation for our 0.13 version. Our choice to move away from Flask and FastAPI as a core part of our library is based on our experience with hundreds of users and use cases

Wait... so you're tell me FastAPI is slow...