Ask HN: What are you using to serve ML models in low latency?

Posted by avin_regmi on Wednesday February 27, 2019 @12:58PM
from the ask-hn dept.

https://panini.ai/ is the easiest and fastest way to serve ML/DL models at low latency and makes the model deployment to Kubernetes in a few minutes. It also handles load balancing, caching and batching of user inputs. What are you guys using to serve ML models in low latency?

( 2 comments | 2 points | on HN )

Re:Ask HN: What are you using to serve ML models in low late... by malux85 on Thursday February 28, 2019 @07:43AM (#19270934)

Low latency for us means we can't spend 100+ ms on a round trip to an external server / hosted solution.

If your unique selling point is low latency, you should at least show some numbers / benchmarks on your homepage.

and finally, there's no way us or our clients would allow our models to be uploaded to an external provider, it would have to be on-prem

[ Parent | on HN ]

Re:Ask HN: What are you using to serve ML models in low late... by avin_regmi on Friday March 08, 2019 @06:13PM (#19343163)

How big is low latency issue for you? What happens if it's more than 100ms? Also we do offer our software to be deployed in your kubernetes couster via helm.

[ Parent | on HN ]