Ask HN: Cheaper way to do model inference?

Posted by masterofall2612 on Friday May 31, 2024 @04:56PM
from the ask-hn dept.

Does anyone know of any solutions for saving GPU compute during server downtime? Is there a managed solution to turn off a pod and turn it back on when I need it? I'm currently doing model inference and most of the time I'm just paying for compute without serving any user requests.

( 3 comments | 2 points | on HN )

Re:Ask HN: Cheaper way to do model inference? by PaulHoule on Friday May 31, 2024 @05:25PM (#40540430)

You are running inference on something like an EC2 instance?

[ Parent | on HN ]

Re:Ask HN: Cheaper way to do model inference? by masterofall2612 on Friday May 31, 2024 @06:20PM (#40540931)

Yea exactly that's what I'm doing right now. I've looked into solutions like RunPod serverless, anyone have any experience with it?

[ Parent | on HN ]

Re:Ask HN: Cheaper way to do model inference? by brianjking on Friday May 31, 2024 @08:58PM (#40541888)

Huggingface Inference Endpoints can autoscale to 0 and cost nothing when not being used.

[ Parent | on HN ]