News for hackers. Stuff that matters.
Re:Show HN: Benchmarks and comparison of LLM AI models and A... by AaronFriel on Tuesday January 16, 2024 @05:02PM (#39019766)
Variance would be good, and I've also seen significant variance on "cold" request patterns, which may correspond to resources scaling up on the backend of providers.

Would be interesting to see request latency and throughput when API calls occur cold (first data point), and once per hour, minute, and per second with the first N samples dropped.

Also, at least with Azure OpenAI, the AI safety features (filtering & annotations) make a significant difference in time to first token.

There are no comments on this comment yet.