Re:Every Way to Get Structured Output from LLMs
by joatmon-snoo on Tuesday June 18, 2024 @04:09AM
(#40715256)
Author here- yes, reprompting can work well enough if the latency hit is acceptable to you.
If you’re driving user-facing interactions with LLMs, though, and you’re already dealing with >1min latency on the first call (as many of our current users are!), waiting for another LLM call to come back is a really frustrating thing to block your UX on.
If you’re driving user-facing interactions with LLMs, though, and you’re already dealing with >1min latency on the first call (as many of our current users are!), waiting for another LLM call to come back is a really frustrating thing to block your UX on.