OpenAI is too cheap to beat

Re:OpenAI is too cheap to beat by eurekin on Thursday October 12, 2023 @02:40PM (#37861169)

Didn't see batching taken into equation, might skew a bit

[ Parent | on HN ]

Re:OpenAI is too cheap to beat by sidnb13 on Thursday October 12, 2023 @02:45PM (#37861249)

Yep, batching is a feature I really wish the OpenAI API had. That and the ability to intelligently cache frequently used prompts. Much easier to achieve this with a hosted OS model, so I guess it's a speed + customizability/cost tradeoff for the time being.

[ Parent | on HN ]

Re:OpenAI is too cheap to beat by advaith08 on Thursday October 12, 2023 @03:19PM (#37861801)

imo they dont have batching because they pack sequences before passing through the model. so a single sequence in a batch on OpenAI might have requests from multiple customers in it

[ Parent | on HN ]

Re:OpenAI is too cheap to beat by sidnb13 on Sunday October 15, 2023 @08:03PM (#37894563)

Ah that would make sense. Similar to vLLM which does dynamic packing.

[ Parent | on HN ]