chaoyu_

Check out BentoML https://github.com/bentoml

Most team don't really concern about the security side of things when using pickle for model deployment, more about performance and resource utilization. It works ok for very light weight models, but doesn't scale.

Regardless of the architecture, you would still need the model server for online serving use cases. And you’re right that it’s best to decouple the validation/preprocessing logic from the model runtime from architecture perspective, and that’s exactly what BentoML does but a regular web framework can not do.

Exactly! A vertical solution focused on model serving and deployment would make more sense if it can easily integrate with the ML training platform for CICD

That’s is true, it would be more accurate to say FastAPI (or any ASGI) framework along is not enough for ML model serving, you need batching and runner for performance. And some additional Ml focused features for integration with other ML tools