How fast can one reasonably expect to get inference on a ~70B model?

Re:How fast can one reasonably expect to get inference on a ... by mks_shuffle on Saturday May 25, 2024 @01:44AM (#40472876)

You can try Groq API for faster inference. They use custom hardware to speed up the inference. Supported open models can be found here: https://console.groq.com/docs/models (includes llama-70b)

[ Parent | on HN ]

Re:How fast can one reasonably expect to get inference on a ... by yungtriggz on Monday May 27, 2024 @02:13PM (#40493141)

thanks, tried this to some mixed results. seems like they have caps on speed/rate limits etc if you havent spoken to them so might reach out

[ Parent | on HN ]