Ask HN: Next steps for scaling a scikit-learn Flask ML API
We currently have an internal API that's core to our business. The models are loaded as .pkl files with scikit-learn joblib and served via Flask w/ Gunicorn using Gevent. We've tried Tornado as a worker class and Cherrypy as a replacement for Gunicorn -- none produce significant performance benefits.We're hosting it in a Kubernetes cluster with really large nodes (140GB). Each container user...