News for hackers. Stuff that matters.
Re:Chatbot Arena Leaderboard: Gemini 1.5 Flash, Pro and Adva... by dilap on Tuesday May 28, 2024 @04:14PM (#40505073)
groq is super fast and cheap, but it seems to have (for normal plans) a very-low 30 requests/minute rate limit; to get a higher-limit, it's "contact us" enterprise pricing (which I haven't done, so I'm not sure what the results will be).

edit: ah, currently it's "free beta access with low rate limits" and "enterprise" that is available, but they are planning to release "on demand pay per token" with higher rate limits soon; exact pricing unknown.

https://pasteboard.co/OnLHjusiv6QT.png

another limitation of groq, which may or may not matter for your use case, is there's no way to pre-fill part of the llm response, which can be a very useful technique (there's also no support for the more general feature of "restrict sampling to a grammar", tho that is a much bigger ask).

There are no comments on this comment yet.