Re:All the Hard Stuff Nobody Talks About When Building Produ...
by ukuina on Saturday May 27, 2023 @10:54PM
(#36100716)
There's no real benefit to streaming if you are planning to use the LLM output downstream (say, in a SQL query). LLM latency is a major annoyance right now, whether locally-hosted or cloud-based.
[ Parent
| on HN ]
There are no comments on this comment yet.