Re:Ask HN: What do you use for ML Hosting
by thundergolfer on Tuesday May 02, 2023 @05:25PM
(#35794435)
> are worth avoiding so you don't get stuck with somebody else's framework.
Modal eng here. Modal is not setup as a framework. Think of more as Python-defined serverless infrastructure that has native support for the Python runtime. This is in some places called "Infrastructure from code", as opposed to "Infrastructure as code" which means just source-controlling K8s YAML and Cloudformation.
A major benefit of this approach is that the cloud becomes part of your dev loop, as opposed to doing `docker build`, `docker push`, `kubectl`, etc just to ship a change to a GPU.
In the script I posted Modal APIs are mixed in with standard Python code for brevity, but many customers just keep their code in their own modules and have a `modal_infra.py` module that defines the serverless infrastructure.
Re:Ask HN: What do you use for ML Hosting
by version_five on Tuesday May 02, 2023 @05:29PM
(#35794479)
Understood, thanks for clarifying. I'll edit my post.
Re:Ask HN: What do you use for ML Hosting
by quickthrower2 on Wednesday May 03, 2023 @08:46AM
(#35801745)
Glad to see this concept being used. I was thinking the other day how odd it is to have IaC then keep the code siloed from the app code. Why doesn’t the app make it’s own infra? I think you have articulated how that would work.
Re:Ask HN: What do you use for ML Hosting
by naderkhalil on Tuesday May 02, 2023 @05:32PM
(#35794513)
That makes sense, Brev.dev is a really simple way to run your code on a configured GPU without having to change your code. It'll also optimize your GPU to save money when possible.
I'm assuming you know what you need for a GPU. If you're unsure, consider trying to run inferences on a CPU and see how long it takes and if it could work.
And then just look at price and reliability for a gpu machine with the different cloud providers. Ovh is cheap but the only thing worse than their reliability is their customer service. Various niche players offering V100s used to pop up that were pretty cheap. AWS is more expensive, more reliable, they may still have availability problems. Paperspace looks pretty good. Etc.